A Lighthouse score of 98 tells you nothing about whether GPTBot can read your homepage. These are now separate questions with separate answers, and the gap between them is the single most consequential architectural blind spot in enterprise web estates built on React or Vue over the past decade. A site can be fast, accessible, and beautifully scored by every human-facing audit tool, while presenting an empty document to the crawlers that feed ChatGPT, Claude and Perplexity. This is not a hypothetical edge case. It is the default outcome of client-side rendering, and most engineering teams have no visibility into it because nothing in their existing tooling checks for it.
The reason this matters now, rather than in some future GEO roadmap item, is that the crawlers in question are already indexing your competitors’ content and citing it in answers your prospects are reading. If your product pages, service descriptions or technical documentation exist only after a JavaScript bundle executes, you are not being deprioritised by these systems — you are architecturally invisible to them. That is a different category of problem to a ranking disadvantage, and it requires a different diagnostic.
The crawler landscape: who renders, who doesn’t
The AI crawler ecosystem is not homogeneous, and treating it as a single entity is the first mistake technical teams make when auditing for GEO. There is a hard split between crawlers built on browser-rendering infrastructure and crawlers that fetch raw HTTP responses and stop there.
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Amazonbot and Meta-ExternalAgent fall into the second category. They issue an HTTP request, receive whatever HTML the server returns at that exact moment, and parse it as text. They do not instantiate a DOM, they do not execute the JavaScript they download, and they do not wait for asynchronous data fetches to resolve. Large-scale empirical analysis of GPTBot fetch logs — sample sizes running into the hundreds of millions — has found no evidence of JavaScript execution whatsoever. The crawler downloads your bundle.js file, but it processes it as a blob of text for training purposes, not as executable code that produces content.
Google-Extended is the exception, and it’s an important one because it explains why teams frequently misdiagnose this problem. Google-Extended rides on the same Web Rendering Service infrastructure as Googlebot — an evergreen Chromium instance that executes JavaScript, waits for network idle, and captures the resulting DOM. AppleBot behaves similarly. So a site can genuinely perform well in Google’s AI Overviews while being simultaneously blind to Claude and Perplexity, and the Search Console data that would normally reassure an engineering lead shows nothing wrong, because Search Console is reporting on the one crawler in the ecosystem that doesn’t have the problem.
This creates what is, in practice, a split-visibility architecture: one rendering pathway for Google’s AI surfaces, and a completely different, unrendered pathway for every other major answer engine. Most enterprise teams are unaware this split exists because their monitoring stack was built around Google’s crawler behaviour and never accounted for the others.
What actually arrives at the crawler’s door
To understand the mechanism precisely, it helps to separate what the server sends from what the browser produces, because these are two distinct artefacts and only one of them is visible to most AI crawlers.
On a client-side rendered React or Vue single-page application, the server’s HTTP response — the thing a non-rendering crawler actually receives — is typically a near-empty shell: a <head> block with basic meta tags, a <body> containing little more than <div id="root"></div> or <div id="app"></div>, and a set of <script> tags referencing JavaScript bundles. There is no heading text, no body copy, no product description, no FAQ content. Everything a human associates with “the page” is generated after that shell loads, when the JavaScript bundle executes in the browser, fetches data, and mutates the DOM. A JavaScript-agnostic crawler never gets past the shell. It has, for all practical purposes, indexed a blank page.
On a server-side rendered or statically generated site — Next.js in SSR or SSG mode, Nuxt in universal mode, or a conventional server-rendered platform like plain WordPress — the situation is structurally different. The HTTP response itself contains the fully composed markup: headings, body text, internal links, and any structured data emitted server-side. There is no dependency on script execution for the content to exist, because the content was already assembled before the response left the server. A crawler that never executes a line of JavaScript still receives everything, because nothing about the content’s presence depends on JavaScript in the first place.
This is the entire crux of the rendering barrier: it is not a performance problem, an accessibility problem, or an SEO problem in the traditional keyword sense. It is a question of where in the request lifecycle your content comes into existence. If it comes into existence in the browser, half the answer-engine ecosystem never sees it.
A reproducible audit: seeing what the crawler sees
Technical teams don’t need to take a vendor’s word for whether this affects them. The methodology is simple enough to run in an afternoon, and it produces a direct, defensible diff between the two realities a page can present.
The baseline test is a raw HTTP fetch using the actual user-agent string of the crawler in question:
curl -A “GPTBot” https://example.com/product-page -o gptbot_response.html
Repeat this with the ClaudeBot, PerplexityBot and Amazonbot user-agent strings. Each will return identical HTML unless the server is doing something unusual, because none of them execute JavaScript, so their treatment of a CSR page will be indistinguishable from one another — a useful sanity check if your results look inconsistent.
Against that, capture what a rendering client sees. The most direct method is a headless browser automation script — Playwright or Puppeteer — that navigates to the same URL, waits for network idle, and extracts the resulting document.documentElement.outerHTML. This is functionally what Google-Extended and a human visitor both receive, and it should be treated as the “full” version of the page for comparison purposes.
With both artefacts captured, a plain text diff between the raw curl response and the rendered DOM extract will show, line for line, exactly which content exists only after script execution. In practice this diff on a CSR-heavy page is not subtle — it is usually the difference between a two-hundred-byte shell and a fully composed document running into tens of kilobytes of markup. There is no ambiguity in the result; either the content is present in the raw response or it isn’t.
A cheaper manual version of the same test, useful for a first-pass sanity check before committing engineering time to the scripted version, is to disable JavaScript execution entirely in browser developer tools and reload the page. Whatever remains visible is, broadly, what a non-rendering AI crawler will receive. It is not a perfect proxy — browsers with JavaScript disabled still parse some things differently to a raw HTTP fetch — but it is fast, requires no tooling, and reliably surfaces the problem when it exists.
For teams that want this running continuously rather than as a one-off audit, the same logic — user-agent-specific fetch versus rendered DOM, diffed automatically — can be built into a scheduled job that flags regressions whenever a deployment reintroduces client-side-only content on a page that was previously server-rendered. This is the kind of check that belongs in CI, not in a quarterly audit, because rendering regressions are usually introduced by well-intentioned frontend changes that nobody thinks to test against a non-executing crawler.
Why this doesn’t stop at content — it breaks discovery too
The rendering barrier has a second-order effect that is easy to miss if the audit stops at a single page: it affects whether a crawler ever discovers other pages on the site at all. If internal navigation is generated by JavaScript — a common pattern in SPA architectures where a router component builds the menu and links dynamically — a non-rendering crawler has no anchor tags to follow in the raw HTML. It cannot discover child pages it wasn’t already aware of through a sitemap or external link.
This has been observed directly: in controlled testing, internal links generated solely via JavaScript were effectively ignored by Meta-ExternalAgent and Amazonbot, while Googlebot — rendering-capable — discovered the same pages without difficulty. The practical consequence is that entire sections of a site can be absent from an AI crawler’s index not because the content itself is bad, but because the crawler never found the door. This is worse than an individual page being unreadable; it’s a structural discovery failure that compounds across every page reachable only through JS-rendered navigation.
The same mechanism silently defeats structured data strategy. JSON-LD schema, canonical tags and meta descriptions injected into the DOM after page load via JavaScript are invisible to the same crawlers that can’t see body content. A site can have meticulously engineered Organization schema, sameAs links and Product markup — and none of it will reach a crawler that stops reading at the empty shell. Entity signalling work is wasted if it’s implemented in the wrong layer of the stack.
Closing the gap without a framework rebuild
The instinct on discovering this is often to treat it as grounds for a full migration to a server-side rendering framework. That is sometimes the right call, but it is rarely the cheapest or fastest one, and for an existing enterprise application it carries real architectural risk that should be weighed against the alternatives rather than assumed as the default fix.
A full SSR migration — moving a large existing CSR application to Next.js SSR, Nuxt universal mode, or Angular Universal — is a genuine rebuild. It typically means refactoring data-fetching patterns, restructuring component lifecycles around server rendering constraints, and accepting that every request, including bot traffic, now carries a server-side compute cost to produce the HTML. For applications with meaningfully dynamic per-user content, this is often the architecturally correct long-term direction. For a large existing enterprise codebase, it is also the most expensive option on the table, and the timeline required makes it unsuitable as an urgent fix.
Static Site Generation is the right answer for a specific and common subset of enterprise content: marketing pages, documentation, blog content, service and product pages that don’t change per-user and don’t need real-time data. Pre-building this content at deploy time — via Next.js’s static generation approach or equivalent tooling in other frameworks — produces exactly the same fully-composed HTML response that SSR gives you, without runtime server compute per request. The implementation cost is moderate: it requires a build pipeline capable of generating and deploying static output, and a workflow for triggering rebuilds when content changes, but it does not require restructuring the application’s runtime architecture.
Dynamic rendering — sometimes called prerendering — is the pragmatic middle path, and for most enterprise teams facing this problem for the first time, it is the correct starting point. The principle is straightforward: detect the requesting user-agent at the server or CDN edge, and if it matches a known bot pattern, route the request to a prerendering layer that returns a fully-rendered HTML snapshot instead of the CSR shell. Human users continue to receive the standard client-side rendered application, unchanged. This can be implemented through a third-party service, or via a self-hosted headless Chrome snapshotting layer that pre-captures and caches rendered versions of key pages.
The infrastructure cost profile here is favourable relative to a rebuild. A well-configured edge-side prerendering layer serves cached snapshots in the low tens of milliseconds, and because it sits in front of the existing application rather than inside it, it adds no additional load to production application servers — the rendering work happens once, at cache-generation time, not on every bot request. The engineering effort is largely confined to user-agent routing logic and cache invalidation strategy, not application refactoring. This makes it the more defensible choice when the business case for AI-crawler visibility needs to be delivered against a near-term deadline rather than a multi-quarter rebuild programme.
The trade-off to be honest about is architectural cleanliness versus speed to fix. Dynamic rendering introduces a second rendering path that must be kept in sync with the primary application — a snapshot that goes stale, or a caching layer misconfigured against a content update cycle, will quietly reintroduce the exact problem it was built to solve. SSR and SSG avoid this by having only one source of truth for the HTML. Enterprise teams with the resource and appetite for a framework-level change should treat prerendering as a bridge, not a permanent architecture; teams without that appetite in the near term should treat it as a legitimate, durable fix, provided the cache invalidation discipline is engineered properly from the outset.
The takeaway for technical decision-makers
The rendering barrier is not a niche technical curiosity — it is a direct, mechanical explanation for why content that performs well in traditional search can be entirely absent from AI-generated answers. It sits upstream of every other GEO consideration: semantic chunking, entity schema, and citation-ready structure are all irrelevant if the crawler consuming your site never receives the HTML those techniques depend on.
The practical discipline this demands is not complicated, but it does need to be deliberate. Run the curl-versus-rendered-DOM diff against your own site using the actual user-agent strings of GPTBot, ClaudeBot, PerplexityBot and Amazonbot before commissioning any GEO work built on top of the current architecture. If the diff shows a meaningful content gap, the fix does not require a framework migration to be credible — a properly engineered dynamic rendering layer, correctly scoped to bot traffic and rigorously cache-invalidated, closes the gap in weeks rather than quarters. What it does require is treating rendering strategy as a first-order architectural decision rather than a frontend implementation detail, because for a growing share of the traffic that determines whether your business gets named in an AI-generated answer, it is the only decision that matters.
Frequently asked questions
Do GPTBot and ClaudeBot execute JavaScript?
No. Large-scale analysis of crawler fetch logs shows GPTBot, ClaudeBot, PerplexityBot, Amazonbot and Meta-ExternalAgent issue a raw HTTP request and parse whatever HTML is returned as text, without instantiating a DOM or running scripts.
Why does a site perform well in Google AI Overviews but not in ChatGPT or Perplexity answers?
Google-Extended runs on the same Chromium-based Web Rendering Service as Googlebot and does execute JavaScript, so it sees fully rendered pages, while every other major AI crawler receives only the unrendered HTML shell — creating a split-visibility architecture invisible to Search Console.
How can we test whether our site is visible to AI crawlers?
Fetch a page with curl using each crawler’s user-agent string, capture the same page’s fully rendered DOM with a headless browser like Playwright, and diff the two outputs; any content missing from the raw fetch is content non-rendering crawlers never see.
Does client-side rendering also block content discovery, not just content reading?
Yes. If internal navigation is built dynamically by a JavaScript router, non-rendering crawlers have no anchor tags to follow in the raw HTML, meaning entire site sections can go undiscovered even when Googlebot indexes them without issue.
