Can AI Crawlers Read Your Site? We Checked 342 Launch Sites Without JavaScript

In October 2026 we requested 365 product sites launched on aat.ee the way a crawler that doesn't run JavaScript would, and counted the text in the HTML that came back. Of the 342 that answered with HTTP 200, 37 (10.8%) returned fewer than 50 words of body text. Almost all of those still had a title and a meta description, so the page looks fine in a link preview while the content itself arrives only after JavaScript runs. When we loaded the same 37 pages in a real browser, 23 of the 30 that finished loading showed 150 words or more.

This matters because the crawlers behind ChatGPT, Claude and Perplexity were observed not to execute JavaScript. If your product description exists only in the rendered page, those systems may never see it. Below are the numbers, the cause, and a check you can run on your own site in two minutes.

Which crawlers run JavaScript

OpenAI, Anthropic, Meta, ByteDance and Perplexity don't document whether their crawlers render JavaScript. The best available evidence is a December 2024 study by Vercel and MERJ. It found that none of the major AI crawlers it examined rendered JavaScript: OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. ChatGPT's and Claude's crawlers did download JavaScript files (11.50% and 23.84% of their requests), but they didn't execute them. Gemini, which uses Googlebot's infrastructure, did render. So did AppleBot.

Googlebot renders JavaScript, but in a separate step. Google's JavaScript SEO guide says pages are queued and "once Google's resources allow, a headless Chromium renders the page and executes the JavaScript." Google still recommends server-side rendering or pre-rendering, because "it makes your website faster for users and crawlers, and not all bots can run JavaScript."

The study is one observation, almost two years old, and crawler behaviour can change without notice. It is still the best data available, and none of the companies has contradicted it.

How we measured

• Sample: 400 project URLs drawn at random from products launched on aat.ee in the 120 days before the audit. We kept one URL per hostname and dropped 14 links that pointed to GitHub, app stores or Hugging Face rather than the product's own site, which left 365 sites. 342 returned HTTP 200 and are the basis of the numbers below.
• Request: one GET per site with a user agent carrying OpenAI's OAI-SearchBot token, following redirects, without executing JavaScript.
• Measure: words of visible body text after removing <script>, <style>, <noscript>, <template> and <svg> elements. We also recorded whether the HTML contained a title, a meta description, a canonical link, JSON-LD and an <h1>.
• Limits: we fetched only the URL the founder submitted, usually the home page. Our requests came from our own network, not OpenAI's IP ranges, so a site that treats verified crawlers differently could serve them something else. Text inside script tags was not counted. The Vercel study notes that data embedded in the initial HTML, such as JSON, "may still be indexed", so for some frameworks our count understates what a crawler gets. Most of the empty pages below are single-page apps whose content lives in separate JavaScript bundles, which the study found these crawlers download but don't run.

The results

The median site returned about 1,180 words, so most launch sites are in good shape. The problem is concentrated: 37 sites, nearly all with almost nothing in the body.

The head is filled in and the body is empty. Of the 37 thin pages, 36 had a <title> and 34 had a meta description. Only 4 had an <h1>, and only 4 had five or more links. A crawler that reads this HTML learns the product's name and a one-line pitch, and nothing about features, pricing or who it is for.

One pattern explains most of them. 30 of the 37 served an empty <div id="root"> or <div id="app"> and built the whole page in the browser, which is the default for a single-page app made with Vite or Create React App. 12 were Vite builds. By contrast, only 1 of the 165 Next.js sites in the sample was thin. Next.js renders pages on the server by default, so the content is in the first response.

Templates leak. 16 of the 37 still had Create React App's default meta description, "Web site created using create-react-app". Those 16 came from three submitter accounts launching one family of similar products, so they inflate the total. Leave them out and the thin share is 21 of 326, or 6.4%. That is still about one launch in sixteen.

In a browser, the content is there

We then loaded the 37 thin pages in headless Chromium, waited five seconds after the load event, and counted the visible text again. 30 pages finished loading. 7 timed out after three attempts.

The median page went from 0 words in the raw HTML to about 475 after rendering. So for most of these sites the copy exists, and a visitor in a browser sees it. It just isn't in the HTML the server sends, which is the only part a non-rendering crawler reads. The five that stayed under 50 words were a login screen, a page that rendered blank and three very short pages. Those are thin for everyone, not only for crawlers.

A side finding: some bot protection blocks the crawler too

Three sites answered our OAI-SearchBot request with HTTP 403 and answered the same request with a Chrome user agent with 200. Two of them returned a Cloudflare challenge (cf-mitigated: challenge). None of their robots.txt files disallowed AI crawlers, so the owners probably did not mean to block them.

We can't tell from this whether OpenAI's real crawler is blocked. Bot protection such as Cloudflare's can recognise verified bots by their IP addresses, and our requests did not come from OpenAI's. Treat it as a reason to look: if you use Cloudflare, check the AI crawler and bot-fight settings in the dashboard and confirm that they match what your robots.txt says.

Check your own site in two minutes

1. Fetch it the way a crawler does.

   UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)'
   curl -sL -A "$UA" -o raw.html -w '%{http_code}\n' https://yoursite.com
   grep -c 'a sentence from your hero section' raw.html

   UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)'
   curl -sL -A "$UA" -o raw.html -w '%{http_code}\n' https://yoursite.com
   grep -c 'a sentence from your hero section' raw.html

   A status other than 200 usually means the crawler gets an error or challenge page instead of your content. A count of 0 means that exact sentence is not in the HTML. Pick a short phrase, since markup can split a long one.

2. Look at the page without JavaScript. In Chrome DevTools, open the command menu (Ctrl+Shift+P or Cmd+Shift+P), run "Disable JavaScript" and reload. What you see is roughly what a non-rendering crawler reads.

3. Check Google separately. Search Console's URL Inspection shows the HTML Googlebot rendered. It tells you about Google only.

4. Repeat for your pricing and feature pages, not just the home page. Those are the pages an assistant needs to describe your product.

How to fix it, by stack

Three rules apply whatever you use:

• Put the facts in the first response. That means the product name, what it does, who it is for, pricing and the main links, in the HTML the server returns.
• Serve bots the same page as people. Google calls dynamic rendering, a separate pre-rendered version for crawlers, "a workaround and not a recommended solution." Microsoft's GEO guide says: "Ensure rendered DOM contains the same facts consumers see — never serve different HTML to bots."
• Don't stop at the <head>. A good title and description help with link previews. They are not enough for a crawler to describe your product.

FAQ

Does Google see my client-rendered content? Usually, eventually. Googlebot queues pages for rendering and runs JavaScript with headless Chromium. That covers Google Search and Gemini. To be a supporting link in AI Overviews, a page "must be indexed and eligible to be shown in Google Search with a snippet", so slow or failed rendering can still cost you there. It does not cover ChatGPT, Claude or Perplexity, whose crawlers were observed not to render.

Is an empty <div id="root"> a ranking penalty? No. Nothing penalizes it. The cost is that a non-rendering crawler has nothing to read, so it can't quote or describe the page.

My site has a good meta description. Isn't that enough? It gives a crawler one sentence. An assistant asked what your product does, or how it compares with an alternative, needs the body copy: features, pricing, use cases.

Do I need to rebuild my app? Usually not. Pre-rendering the public pages, such as the home, pricing, feature and docs pages, is enough. The logged-in app can stay client-rendered.

Will fixing this get me cited by ChatGPT? Nobody can promise that. Being readable is a precondition. Whether you are cited depends on what other sources say about you and how well your page answers the question. Our AI search optimization playbook covers the rest.

Related reading

• The 2026 founder's guide to AI crawlers: robots.txt, llms.txt and citation
• Reading AI crawler logs: what GPTBot actually requests
• The website launch checklist
• 7 GEO claims, checked against Google's own documentation

Sources

• Vercel and MERJ: The rise of the AI crawler (December 2024)
• Google Search Central: Understand JavaScript SEO basics
• Google Search Central: Dynamic rendering as a workaround
• Google Search Central: AI features and your website
• Next.js: Server and Client Components
• Microsoft Advertising: From discovery to influence: a guide to GEO
• aat.ee: raw-HTML audit of 365 launched product sites (342 analysed), October 2026

Every product on aat.ee gets a permanent, server-rendered listing page, so crawlers can read a description of your product even before your own site is ready for them. Submit your product, or see the listing options.