For the past eighteen months, the industry answer to “how do we know AI crawler traffic is real?” has been forensic: pull the logs, cross-reference the IP range against the published list, check whether the request pattern matches known bot behaviour, write it up after the fact. That approach was always a compromise. IP ranges get reassigned, published lists lag behind infrastructure changes, and a user-agent string is just text a client chooses to send. Anyone can set it to GPTBot. The entire model rested on trust in a label, verified retrospectively.
That compromise is now becoming untenable, for a specific reason: the traffic hitting enterprise infrastructure is no longer only read-only. WebMCP and equivalent agentic protocols expose write access — form submission, transaction initiation, account modification — to the same category of client that, until recently, only ever asked to read a page. Forensic log analysis is an adequate control for a crawler that might waste some compute indexing your pricing page. It is not an adequate control for an agent that might submit a purchase order. This is the point at which bot identification has to stop being a classification exercise performed by a human reviewing a report, and start being a real-time access-control decision performed by infrastructure, before the request is served.
Why String-Based Identification Was Always Provisional
The underlying weakness in user-agent and IP-based verification is that both signals are presented by the client, not proven by the client. A user-agent string is a claim. An IP address is a network-layer artefact that can be routed through proxies, shared hosting, or infrastructure the crawler operator doesn’t fully control day to day. Enterprises that have built GPTBot or ClaudeBot verification into their monitoring stack — checking published IP ranges, doing reverse DNS lookups — have built a reasonably good detector of accidental misclassification. They have not built a detector of deliberate impersonation, because nothing in that stack requires the requester to prove possession of anything secret.
This distinction matters because the incentive to impersonate a legitimate AI crawler is growing in direct proportion to how much weight enterprises are putting on AI-referred traffic and agentic transactions. A scraper that wants unrestricted access to gated content has every reason to declare itself GPTBot. A malicious actor probing for exploitable WebMCP endpoints has every reason to look, at the network layer, exactly like a sanctioned agent. IP allowlisting and user-agent sniffing were built for a web where the worst outcome of misidentification was a slightly noisy analytics dashboard. They were not built for a web where misidentification can mean granting write access to a system that executes actions.
The Mechanism: What RFC 9421 Actually Proves
HTTP Message Signatures, standardised as RFC 9421 in February 2024, solves a narrower and more useful problem than TLS does. TLS secures the connection between two endpoints — it tells you the channel is encrypted and that you’re talking to the server whose certificate you checked. It says nothing about who is on the other end of that channel once intermediaries, proxies, or CDNs are involved, and it says nothing that survives the request being logged, replayed, or inspected later. RFC 9421 operates at the message level instead. It lets a client cryptographically sign specific components of an HTTP request — the method, the target URI, the authority header, a timestamp, and critically, a SHA-256 digest of the request body via RFC 9530 — and attach that signature directly to the message via two headers: Signature-Input and Signature.
The mechanical sequence is straightforward once you separate it from the surrounding standards-track jargon. The signer decides which components of the message matter for this context and constructs a canonical string from them — a defined, repeatable serialisation that survives incidental transformations like header reordering or capitalisation changes introduced by intermediaries. That canonical string is what actually gets signed, using a private key — typically Ed25519 for its speed and small signature size, though ECDSA P-256 is also supported. The signature, the key identifier, the algorithm, a creation timestamp, and a nonce for replay protection are packaged into the signature headers and sent alongside the original request. On the receiving end, the verifier reconstructs the same canonical string from the request it actually received, retrieves the public key associated with the stated key identifier — usually from a published JWKS directory at a well-known URI — and checks whether the signature validates against that reconstructed string.
The practical consequence is that the request proves its own origin and its own integrity, independent of which network path it travelled. Nobody in the middle — a proxy, a CDN, a load balancer — can alter the method, the URI, or the body content without invalidating the signature. This is why the mechanism generalises so well from simple read requests to WebMCP-style write operations: because the body digest is part of what gets signed, an attacker who intercepts a legitimate signed request cannot silently change a POST into a DELETE, or alter a transaction payload, without the tampering being immediately detectable at verification time.
Web Bot Auth: Turning the Primitive Into a Web-Wide Protocol
RFC 9421 is a general-purpose signing mechanism; it doesn’t by itself define how bots should identify themselves to arbitrary websites at scale. That’s the gap the IETF Web Bot Auth draft — formally draft-meunier-web-bot-auth-architecture — is closing. The draft reached a standards-track specification milestone in April 2026, with an accompanying best-current-practice document targeted for August 2026, following an IETF working group chartered earlier in the year. Its contribution is a convention layered on top of RFC 9421: a Signature-Agent header that points verifiers to a published JWKS directory, so a website doesn’t need a pre-negotiated relationship with every bot operator on the internet to verify their signatures — it can fetch the operator’s public keys on demand from a well-known location and cache them.
Despite its formal status still being a draft, production deployment has moved faster than the standards process. Cloudflare has built Web Bot Auth verification directly into its Verified Bots programme, extending it beyond the older, weaker signal of “this IP range matches a published list.” AWS added Web Bot Auth support to WAF Bot Control (version 4.0 and later) in November 2025, with the WAF regularly polling bot operators’ key directories, validating signatures automatically, and attaching a verification label — such as web_bot_auth:verified — to the request context, which downstream rules can then act on. Amazon Bedrock AgentCore Browser can sign outbound agent requests automatically, aimed at reducing the CAPTCHA friction that currently makes agentic browsing unreliable. Vercel and HUMAN have added verification support to their respective bot-management layers. The pattern across all of these is consistent: the verification decision has moved from the origin server, after the fact, to the edge, before the request completes.
Who Is Actually Signing Today
Adoption is real but uneven, and the unevenness is the operational detail enterprise teams need to plan around rather than assume away. OpenAI has adopted Web Bot Auth for Operator, its agent product that takes actions on a user’s behalf — a sensible starting point, since Operator is precisely the category of client where identity verification matters most. Anthropic’s ClaudeBot family is reported, per ecosystem documentation and vendor bot directories such as Cloudflare’s, to sign its requests, though as of August 2026 its public key directory was not independently reachable at the standard well-known path — meaning verification in practice currently leans on vendor-curated bot registries rather than direct JWKS resolution, a distinction that matters if your infrastructure does its own verification rather than delegating to a CDN’s bot-management product.
Google’s position is the clearest illustration of why a mixed verification model remains necessary. As of mid-2026, Google’s own guidance is to continue relying on IP address and reverse DNS checks alongside Web Bot Auth, because its primary indexing crawler, Googlebot, is not signing requests at all — only the separate AI-browsing Google-Agent client uses the protocol. An enterprise that switched to signature-only verification and dropped IP-based checks entirely would, today, silently lose the ability to distinguish legitimate Googlebot traffic from anything else. Perplexity, meanwhile, authenticates its Agent API through conventional API keys rather than RFC 9421 signatures, and its user-triggered fetch agent continues to rely on address-based identification. The practical implication is that no enterprise can responsibly deploy signature verification as a single gate in 2026. It has to sit alongside the older IP and behavioural checks as a stronger signal for the operators who support it, not a replacement for the operators who don’t yet.
Authentication Is Not Authorisation — The Gap WebMCP Exposes
This is the distinction that separates a genuinely defensible architecture from a false sense of security. A validated Web Bot Auth signature answers one question precisely: is this request cryptographically attributable to the private key associated with this stated identity? It does not answer the question that actually matters for a WebMCP endpoint: should this identity, having been correctly verified, be permitted to perform this specific action? Those are separate decisions, made at separate layers, and conflating them is the most common architectural mistake likely to emerge as agentic traffic becomes routine.
A verified identity is still just an identity. Knowing with cryptographic certainty that a request originated from OpenAI’s Operator infrastructure tells you nothing about whether the specific action requested — modify a booking, initiate a refund, submit a procurement form — is one your organisation wants that agent, on behalf of that user, to be able to execute unattended. The correct architecture treats signature verification as a gate at the edge that filters out spoofed and unattributable traffic, and treats everything past that gate as still subject to an explicit authorisation policy: scoped credentials that grant only the specific actions a given integration needs, least-privilege access to underlying systems rather than blanket API access, and for higher-risk actions, a human-in-the-loop approval step where the proposed action is surfaced to a human before execution, with the resulting approval itself cryptographically bound to the transaction it authorised. The body-digest property of RFC 9421 is what makes that binding meaningful — it prevents an approved “read this record” signature from being silently reused to justify a “write to this record” action, because the signed content includes the actual payload, not just an abstract claim of identity.
Building This Into the Edge Stack
In a typical enterprise deployment, verification belongs at the CDN or WAF layer, not buried in application code at the origin. The mechanics are consistent across implementations: the edge service parses the incoming Signature, Signature-Input, and Signature-Agent headers, resolves the relevant public key — either from a cached copy of the operator’s JWKS directory or from a vendor-maintained bot registry — reconstructs the canonical signature base from the request as received, and checks it against the claimed signature. Cloudflare Workers can perform this verification directly using the Web Crypto API, which is useful for organisations that want custom policy logic rather than relying entirely on a vendor’s default bot-management ruleset. AWS WAF’s approach of attaching a verification label to the request context, rather than making a binary allow/block decision itself, is architecturally the right pattern: it separates the cryptographic verification step from the business policy decision, letting downstream rules — rate limits, scope restrictions, routing to a sandboxed environment for write actions — apply differently depending on whether the label is present.
The operational payoff is that verified traffic and unverified traffic no longer need to be treated identically by default. An enterprise can afford to be more permissive — higher rate limits, access to more content, eligibility for WebMCP endpoints at all — for traffic that has proven its identity, and correspondingly more restrictive for everything else, including traffic that presents a familiar user-agent string but fails signature verification. That’s a meaningfully different security posture from today’s typical setup, where a spoofed user-agent and a genuine one are functionally indistinguishable until someone reviews the logs.
What This Doesn’t Solve
Cryptographic verification removes a specific class of risk — casual impersonation via a copied user-agent string — without removing several others that enterprise security teams still have to design against. Private key compromise remains the central failure mode: if an operator’s signing key leaks, an attacker inherits the ability to forge legitimate-looking signed requests until the key is rotated and the compromise detected, which means key rotation cadence and revocation mechanisms matter as much as the signing itself. The nonce field in RFC 9421 mitigates simple replay attacks, but only if verifiers actually check nonce uniqueness rather than treating a valid signature as sufficient on its own. And because adoption is partial — Googlebot unsigned, Perplexity’s crawler on API keys, an unknown proportion of smaller AI tools not participating at all — signature verification cannot yet be the sole gate; it has to operate as an additional, stronger signal layered over the IP and behavioural checks that already exist, with scoping and rate-limiting doing the work that identity verification alone cannot.
That last point is the one worth holding onto. Signature-based authentication raises the cost and difficulty of impersonation substantially, but it does not eliminate the need for the compensating controls — least-privilege scoping, human approval for consequential write actions, rate-limiting tiered by verification status — that any competent access-control design already requires. What it changes is the quality of the input those controls are acting on: a real-time, cryptographically attributable identity rather than a self-reported label that could belong to anyone.
The Practical Takeaway
For enterprises currently treating AI crawler auditing as a quarterly log-review exercise, the direction of travel is unambiguous: verification is moving to the edge, and it’s moving there because the traffic it’s verifying is starting to carry write access, not just read requests. The organisations that get ahead of this will implement Web Bot Auth verification at the CDN or WAF layer now, while adoption is partial, rather than waiting for universal support — because the architecture required to handle a mixed environment of signed and unsigned legitimate traffic is more complex than the architecture required once signing becomes the default, and it’s better to build that complexity deliberately than retrofit it under pressure. Separately, and just as importantly, any WebMCP endpoint exposing write or transactional scope needs an authorisation layer that assumes signature verification has already happened and asks the next question anyway: given that we know who this is, what, precisely, should they be allowed to do?
Frequently asked questions
Why is IP address and user-agent based bot verification no longer sufficient?
Both signals are claims presented by the client rather than proven facts — IP ranges get reassigned and a user-agent string can be set to anything — so neither detects deliberate impersonation, which becomes critical once agentic protocols like WebMCP expose write access such as form submission and transaction initiation.
What does RFC 9421 actually prove about an HTTP request?
RFC 9421 (HTTP Message Signatures) lets a client cryptographically sign specific request components, including the method, target URI, authority, timestamp, and a SHA-256 body digest via RFC 9530, so the request proves its own origin and integrity regardless of the network path, and any tampering by an intermediary invalidates the signature.
How does Web Bot Auth extend RFC 9421 for use across the web?
The IETF Web Bot Auth draft adds a Signature-Agent header that points verifiers to a bot operator’s published JWKS directory, letting a website fetch and cache public keys on demand instead of requiring a pre-negotiated relationship with every bot operator.
Which AI crawlers currently sign their requests using Web Bot Auth?
Adoption is uneven: OpenAI’s Operator signs via Web Bot Auth, Anthropic’s ClaudeBot is reported to sign but its key directory was not independently reachable at the standard path as of August 2026, Google’s Googlebot does not sign at all while its separate Google-Agent client does, and Perplexity authenticates via conventional API keys rather than RFC 9421 signatures.
