Every GEO engagement we open begins with the same unexamined assumption: that the AI crawlers a client believes are visiting their infrastructure are the ones actually visiting it. Teams commission content restructuring, entity schema, llms.txt files, semantic chunking — all before establishing whether GPTBot, ClaudeBot or PerplexityBot can reach the pages in question, or whether the traffic logged under those names is genuine at all. This is a sequencing error, and it is an expensive one. Before any GEO investment decision is defensible, the server logs need to be read properly.
The reason this matters now, specifically, is that user-agent strings carry no authentication. A request that arrives claiming to be GPTBot is making an assertion, not presenting a credential. Recent analysis of AI crawler traffic found that a measurable proportion of visits claiming an AI brand identity arrived from IP infrastructure that brand does not operate — impostor traffic, scraper tooling, or competitive intelligence services wearing a familiar mask. If your GEO reporting counts every request bearing the string “GPTBot” as a genuine OpenAI ingestion event, you are measuring noise and calling it signal. That has direct consequences: budget gets allocated against a crawl pattern that may be one-third fabricated, and genuine blocking or throttling issues go undiagnosed because the aggregate numbers look healthy.
Why the User-Agent String Cannot Be Trusted Alone
The user-agent header is a plaintext field set entirely by the client. Any scraper, monitoring tool, or malicious actor can set it to GPTBot, ClaudeBot, or PerplexityBot/1.0 with zero technical barrier. This is not a theoretical vulnerability specific to AI crawlers — it is the same weakness that has existed in HTTP since the protocol’s inception, and the same reason Googlebot verification has required IP-based confirmation for over a decade. What has changed is the incentive structure. AI crawler traffic is now something competitors, data brokers, and content aggregators have reason to impersonate, either to obscure their own scraping activity behind a more palatable identity or to probe which content a target site is exposing to AI ingestion pipelines.
The published IP ranges for legitimate crawlers give you the first half of a verification method. OpenAI publishes GPTBot’s IP ranges via platform.openai.com. Perplexity publishes theirs at a dedicated JSON endpoint, perplexitybot.json, which is machine-readable and intended for automated ingestion into firewall rules. Applebot-Extended’s ranges are published as an extension of the primary Applebot infrastructure. Google-Extended is not a separate crawler with its own IP allocation — it is a product token carried within standard Googlebot requests, which means verifying it requires the same Googlebot IP verification process Google has documented for years, not a separate lookup.
Anthropic’s ClaudeBot is the least straightforward of the group. At various points, engineering teams auditing their logs have found no consistently published IP range or reliable reverse-DNS profile for ClaudeBot traffic, which means a meaningful share of traffic claiming to be ClaudeBot cannot currently be positively confirmed and should be logged as unverifiable rather than assumed genuine in either direction. This is an important distinction: unverifiable is not the same as fraudulent, but it should not be reported to stakeholders as confirmed AI ingestion either.
Forward-Confirmed Reverse DNS: The Verification Mechanism That Actually Works
IP range matching alone has a maintenance problem — published ranges drift, and vendors update them without a synchronised notification mechanism. A more durable verification method, and the one we run as standard on client infrastructure, is forward-confirmed reverse DNS (FCrDNS). This is the same technique used to verify Googlebot’s identity, applied to the newer crawler population.
The method runs in four steps. First, extract the source IP from the raw access log for the request in question. Second, perform a reverse DNS (PTR) lookup on that IP to obtain a hostname. Third, check whether that hostname belongs to the domain the crawler claims to represent — a genuine Googlebot request should resolve to something under googlebot.com, for instance. Fourth, and this is the step teams most often skip, perform a forward DNS lookup on that hostname and confirm it resolves back to the original source IP. If the forward lookup returns a different address, or the reverse lookup fails to resolve at all, the request is not genuine — regardless of what the user-agent string claims.
This four-step loop closes the spoofing gap because it requires control of both DNS directions for the claimed domain, which an impersonator without access to the vendor’s DNS infrastructure cannot fabricate. It is more expensive computationally than a static IP-range match, which is why it is typically run as a batch process against log samples rather than inline on every request, but for periodic audit purposes — monthly or quarterly, ahead of any GEO investment review — it is the standard that should be applied before any crawl statistic is trusted.
An emerging alternative worth tracking is HTTP Message Signatures under RFC 9421, where a crawler cryptographically signs each request with a private key and publishes the corresponding public key for verification. This binds identity to cryptography rather than network topology, which sidesteps the entire IP-drift problem. It remains in experimental preview in server tooling such as mod_pagespeed as of mid-2026, and is not yet something you can rely on as a primary verification layer, but it is the direction this is heading and worth budgeting engineering attention toward over the next eighteen months.
What Legitimate Crawl Behaviour Actually Looks Like
Once verification is in place, the second diagnostic question is whether the crawl pattern itself looks like a genuine AI ingestion agent or like something else wearing the right name. Legitimate AI crawlers exhibit behaviour that is meaningfully different from both human traffic and traditional search engine crawling, and understanding that difference is what lets you read a log file with confidence rather than guesswork.
The most striking characteristic is the crawl-to-referral ratio — the number of pages a crawler requests relative to the number of actual user visits it subsequently sends back to the site. Traditional search crawlers historically operate at ratios in the low double digits; Cloudflare data has put major search crawler ratios around 14:1. AI training crawlers operate on an entirely different scale. ClaudeBot’s ratio has been observed in the tens of thousands to one, and in earlier measurement periods reached figures in the hundreds of thousands to one before improving following the introduction of Claude’s web search product. This is a structural feature of training-oriented crawlers, not a fault: GPTBot and similar agents are harvesting content for model training, not sending users anywhere, so a near-total absence of referral traffic alongside heavy crawl volume is not itself evidence of a problem — it is evidence you are dealing with a training crawler rather than a search-oriented one.
This matters for how you interpret the data. If a client is expecting AI-driven referral traffic to materialise from GPTBot crawl volume, that expectation is based on a misunderstanding of what GPTBot does. Purpose-driven distinction matters here: training bots such as GPTBot and CCBot are not going to send visitors; search-oriented agents such as PerplexityBot and ChatGPT-User carry a much stronger referral incentive because they are retrieving content specifically to answer a live user query. Auditing crawl logs without separating these two categories produces a flattened, misleading picture — you cannot judge PerplexityBot’s performance by GPTBot’s referral behaviour, or vice versa.
The second behavioural marker worth tracking is what these crawlers actually request. Legitimate AI crawlers tend to hit semantic infrastructure aggressively — sitemap.xml, llms.txt where present, JSON-LD structured data blocks — in a pattern that looks more like they are mapping the knowledge structure of the site than browsing it page by page in sequence. New content is often discovered within hours of a sitemap update, which is a useful baseline: if you push new content and see no crawler activity against it within a reasonable window, that is a discoverability signal worth investigating before assuming the crawler simply hasn’t got round to it yet.
Where Robots.txt, llms.txt and the WAF Actually Interact
A recurring failure mode we encounter on enterprise infrastructure is teams treating robots.txt as an enforcement mechanism when it functions as a request. Many AI crawlers, particularly those built for training ingestion, do not reliably respect robots.txt directives, including crawl-delay settings intended to throttle request rate. This is not a compliance failure on the crawler’s part in the same sense a search engine ignoring robots.txt would be — the incentive structures around training data acquisition are simply different, and disallow directives function more as a documented preference than a binding rule for a subset of these agents.
llms.txt sits in a different category entirely and should not be confused with an access control file. It is a proposed, still-evolving convention for signalling to language models what content exists and how it should be interpreted — a guide for AI-facing entry points, not a gate. Treating it as a security or blocking mechanism is a category error that leaves teams with a false sense of control.
The layer that actually enforces access is the WAF or CDN configuration, because it operates at the network edge before the request ever reaches the application layer where robots.txt would be parsed. This is precisely where the most common unintentional blocking occurs. Rate-limiting rules configured to catch scraper abuse or bot floods frequently catch legitimate AI crawlers in the same net, because from a raw traffic-pattern perspective a training crawler making thousands of rapid sequential requests looks structurally similar to a malicious scraping run. Surveys of UK and US websites in early 2026 found that roughly a quarter were unintentionally blocking major LLM crawlers — and the cause was overwhelmingly upstream CDN or WAF configuration rather than a deliberate robots.txt exclusion. This is the audit finding that most often surprises clients: they believe they have opted in to AI visibility by publishing schema and an llms.txt file, while their CDN’s default bot-mitigation ruleset is quietly returning 403s to the very crawlers that content was built for.
The practical implication is that a GEO audit cannot stop at the application layer. It has to trace a request’s full path — CDN edge rule, WAF challenge logic, then robots.txt, then application response — because a block at any one of those layers produces the same symptom in aggregate analytics: an AI crawler that appears to have visited but never actually retrieved content.
Reading the Log Fields That Tell You What Actually Happened
Once verification and access-path tracing are in place, the raw log fields themselves need systematic interpretation rather than a glance at status codes in isolation. Five fields do the diagnostic work.
- Status code. A 200 response confirms full content delivery. A 304 indicates the crawler already held a cached copy and the server confirmed no update was needed — not a failure, but worth tracking if it persists across content revisions, since it may mean cache headers are preventing re-ingestion of updated material. A 403 is an explicit block, worth cross-referencing immediately against WAF logs. A 404 against a URL the crawler should have found via sitemap indicates crawl waste or a structural discoverability problem. Any 5xx response points to server strain under crawl load, which itself can suppress future crawl frequency if the crawler’s own retry logic deprioritises unreliable sources.
- Response time. Elevated latency on crawler requests specifically, compared to human traffic on the same endpoints, suggests the origin is struggling under concurrent crawl volume — a pattern that compounds under aggressive training-bot crawling and can trigger rate-limiting further up the stack, creating a feedback loop where legitimate crawlers get progressively throttled.
- Bytes served. This is the field most teams skip and the one that catches silent degradation. A response returning 200 with a byte count far below the known page size usually means the crawler received a stripped, cached, or partially rendered version of the content rather than the full page — a failure mode that produces a clean-looking log line while actually starving the crawler of the content it came for.
- Requested URL and timestamp together. Plotted against each other, these reveal crawl depth and frequency patterns — whether a crawler is systematically working through the site’s structure or repeatedly hitting a narrow set of entry points, which can indicate either a shallow sitemap or an internal linking structure that isn’t surfacing deeper content.
None of this shows up in GA4 or equivalent client-side analytics, because AI crawlers overwhelmingly do not execute JavaScript. A crawler can retrieve, parse, and cite your content while remaining completely invisible to a JavaScript-dependent tag manager. Server-side log analysis is not an optional supplement to conventional analytics for this purpose — it is the only place this data exists at all.
Tooling That Fits a Mid-Market Budget
None of the above requires enterprise SIEM spend. GoAccess is free, open-source, processes Apache, Nginx, or CDN edge logs directly, and will surface aggressive hosts and user-agent patterns in a terminal dashboard or static HTML report within minutes of pointing it at a log file — sufficient for a baseline audit on most mid-sized infrastructure. For teams needing longer retention and more structured querying across larger log volumes, the ELK stack (Elasticsearch, Logstash, Kibana) provides that capability, at the cost of a heavier setup and maintenance overhead that needs a dedicated engineering owner.
Where infrastructure already sits behind Cloudflare, Bot Management and Radar provide meaningful additional visibility — behavioural fingerprinting and bot scoring even on lower-tier plans, and aggregate crawl-to-referral benchmarking via Radar’s public data, which is useful for sanity-checking whether your own site’s ratios sit within the expected range for a given crawler. Dedicated verification services, and vendor-published tools such as Perplexity’s own bot-verification API, offer a fast secondary check against individual IPs without building the FCrDNS pipeline from scratch.
The right stack depends on scale, but the principle doesn’t: a monthly log audit combining IP/FCrDNS verification, crawl-ratio benchmarking by crawler category, and a full status-code and byte-count breakdown is achievable with open-source tooling and a few hours of engineering time. It does not require a security operations budget.
The Takeaway
Content strategy for AI visibility is only as good as the infrastructure’s proven capacity to deliver that content to genuine AI crawlers. Every recommendation in the GEO literature — semantic chunking, entity schema, llms.txt structuring — assumes the crawler can reach the page and receive an unthrottled, unblocked, full-byte response. That assumption is frequently false, and it is false in ways that don’t show up until someone reads the raw logs rather than the aggregate dashboard. Before commissioning content work against AI visibility, verify the crawlers claiming to visit are genuine, confirm what they’re actually retrieving once verified, and trace the full request path through CDN, WAF, and application layer to rule out unintentional throttling. The content strategy can wait a week. The audit cannot be skipped without the entire investment resting on an unverified assumption.
Frequently asked questions
Why can’t the user-agent string be trusted to identify AI crawlers?
The user-agent header is a plaintext field set entirely by the client, so any scraper or malicious actor can label a request as GPTBot, ClaudeBot or PerplexityBot with no technical barrier; a measurable proportion of traffic claiming these identities has been found arriving from IP infrastructure the named vendor does not operate.
What is forward-confirmed reverse DNS and why does it matter for AI crawler audits?
FCrDNS is a four-step verification method that performs a reverse DNS lookup on the source IP, confirms the resulting hostname belongs to the claimed vendor’s domain, then performs a forward lookup to confirm it resolves back to the original IP; it closes the spoofing gap because an impersonator cannot fabricate control of both DNS directions.
Why doesn’t GPTBot crawl volume translate into referral traffic?
GPTBot and similar training-oriented crawlers harvest content for model training rather than answering live queries, producing crawl-to-referral ratios in the tens of thousands to one, whereas search-oriented agents like PerplexityBot carry a much stronger referral incentive because they retrieve content to answer a specific user request.
Can robots.txt reliably block AI crawlers from enterprise infrastructure?
No; robots.txt functions as a request rather than an enforcement mechanism, and many AI crawlers built for training ingestion do not reliably respect its directives, which is why verification and WAF-level controls need to be assessed separately.
