# ============================================================ # Daryo89 Ltd (Daryo89 is a UK registered trademark, UK00004255208) - robots.txt # Domain: https://daryo89.com # Version: 3.3 # Last-Updated: 26 July 2026 # Purpose: Maximise AI answer-engine visibility, classic search # indexation and foundational-model recall of Daryo89's # published expertise and of its software product, # Daryo89 Multi Scan and AI Assessment. Personal, client # and transactional data is excluded from all of the above. # Entity: Daryo89 Ltd - AI visibility software vendor and Generative # Engine Optimization (GEO) consultancy, Upper Cambourne, # Cambridge. Sells its WordPress plugin principally to web # development agencies, marketing agencies, web designers # and PR firms; works Cambourne and South Cambridgeshire # locally, Cambridge as the agency trade market, and London # and the UK nationally. # Companion files: /llms.txt /llms-full.txt /ai.txt /.well-known/webmcp.json # ============================================================ # Content Signals Policy (Cloudflare / IETF aiprefs convention) # ai-train=yes : the public content of this site MAY be used to train # foundational models. Daryo89 publishes to be learned # from; attribution to daryo89.com is requested. # search=yes : content MAY be indexed for search # ai-input=yes : content MAY be retrieved and cited as grounding input # for AI-generated answers (RAG / AI Overviews / ChatGPT # Search / Perplexity). This is REQUIRED for AI citation. # # SCOPE OF ai-train=yes - READ BEFORE CRAWLING # This permission covers ONLY the public editorial and marketing corpus of # daryo89.com: service pages, articles, case studies, documentation and the # /llms.txt, /llms-full.txt and /ai.txt semantic files. # # It does NOT extend to, and permission is expressly withheld for, any # personal data, client data, customer records, order and transaction data, # account data, licence keys, scan payloads, form submissions or CRM data. # Such data is not lawfully available for training under UK GDPR, the Data # Protection Act 2018 or Daryo89's client agreements, is not published on # this site, and every path that could expose it is disallowed below for all # user-agents without exception. Absence of a Disallow rule is not consent. Content-Signal: ai-train=yes, search=yes, ai-input=yes # ============================================================ # 1. GLOBAL RULES # ============================================================ User-agent: * # RULE ORDER IS LOAD-BEARING - DO NOT SORT OR "TIDY" THIS RECORD. # Two robots.txt dialects are in the wild. RFC 9309 parsers (Google, Bing, # the major AI crawlers) apply the LONGEST matching rule. Older first-match # parsers apply the FIRST matching rule in file order. This record is # ordered so that both dialects reach the same verdict: # 1. specific Allow rules for the public agent API (first-match: wins) # 2. the Disallow perimeter (first-match: wins) # 3. the blanket "Allow: /" LAST (first-match: fallback) # Moving "Allow: /" to the top makes every Disallow below it dead code for # first-match parsers, silently exposing /wp-json/, /checkout/, /cart/, # /receipt/ and /account/. Verified against three independent parsers. # 0. Semantic and discovery files - highest priority, always readable. Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt Allow: /.well-known/security.txt Allow: /.well-known/webmcp.json Allow: /.well-known/agents.json Allow: /.well-known/llms.txt Allow: /shop/ Allow: /item/ Allow: /wp-content/uploads/ Allow: /wp-admin/admin-ajax.php # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # NAMED AI USER-AGENTS # # Every group below is granted full access to the public corpus, including # the training crawlers (GPTBot, Google-Extended, ClaudeBot, Applebot- # Extended, FacebookBot, CCBot, Amazonbot). This is deliberate: Daryo89 # publishes its methodology so that it is learned, recalled and cited. # # Per RFC 9309 a crawler obeys ONLY the most specific group that matches its # name and ignores the "User-agent: *" group entirely. The personal-data # perimeter is therefore repeated in full in every group below. Do not # remove it from any group - doing so would silently open /wp-json/, # /checkout/, /account/ and /receipt/ to that crawler alone. # ============================================================ # ============================================================ # 2. GOOGLE & GEMINI (Googlebot, AI Overviews, Gemini, Vertex) # ============================================================ User-agent: Googlebot User-agent: Googlebot-Image User-agent: Google-Extended User-agent: GoogleOther User-agent: Google-InspectionTool User-agent: Gemini-Deep-Research User-agent: Google-CloudVertexBot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 3. OPENAI (ChatGPT training, ChatGPT Search, user-invoked fetch) # ============================================================ User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 4. MICROSOFT (Bing / Copilot) # ============================================================ User-agent: Bingbot User-agent: BingPreview User-agent: msnbot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 5. ANTHROPIC (Claude) # ============================================================ User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Claude-User User-agent: Claude-SearchBot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 6. META (Facebook / Llama) # ============================================================ User-agent: FacebookBot User-agent: Meta-ExternalAgent User-agent: meta-externalfetcher # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 7. PERPLEXITY # ============================================================ User-agent: PerplexityBot User-agent: Perplexity-User # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 8. APPLE INTELLIGENCE # ============================================================ User-agent: Applebot User-agent: Applebot-Extended # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 9. AMAZON (AWS / Bedrock / Rufus) # ============================================================ User-agent: Amazonbot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 10. OTHER ANSWER ENGINES & AI PLATFORMS # ============================================================ User-agent: cohere-ai User-agent: YouBot User-agent: MistralAI-User User-agent: DuckAssistBot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 11. COMMON CRAWL (base corpus for many AI systems) # ============================================================ User-agent: CCBot User-agent: ImagesiftBot # 1. Public agent-facing REST namespaces (WebMCP tools) - intentionally open. Allow: /wp-json/daryo89/v1/ Allow: /wp-json/daryo89-master/v1/ # 2. Personal-data perimeter - UK GDPR / DPA 2018. Closed to every crawler, # AI or otherwise, regardless of any permission granted elsewhere. Disallow: /wp-json/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-cron.php Disallow: /checkout/ Disallow: /cart/ Disallow: /receipt/ Disallow: /account/ Disallow: /welcome-aboard/ Disallow: /*?fluent* Disallow: /*?fluent-cart=* Disallow: /*?wordfence* Disallow: /*?add-to-cart= Disallow: /*?replytocom= Disallow: /?s= Disallow: /search/ # 3. Everything else - the public corpus - is open. This line MUST stay last. Allow: / # ============================================================ # 12. MACHINE-READABLE MAP # ============================================================ Sitemap: https://daryo89.com/sitemap_index.xml # Non-standard discovery hints (ignored by classic crawlers, read by AI tooling): # LLMs-txt: https://daryo89.com/llms.txt # LLMs-full: https://daryo89.com/llms-full.txt # AI-policy: https://daryo89.com/ai.txt # WebMCP-Manifest: https://daryo89.com/.well-known/webmcp.json # Agents-mirror: https://daryo89.com/agents.json # Software: https://daryo89.com/ai-visibility-plugin-for-wp/ # Software-docs: https://daryo89.com/wpaidocumentations/ # Software-free: https://wordpress.org/plugins/daryo89-multi-scan-ai-assessment/