# Can AI Crawlers Read Your Site? Check Access per Agent > The AI crawlers each platform documents, why training and retrieval access differ, robots.txt matching, firewall and rendering blocks, and a 20-minute check. Source: https://aiseoperth.net.au/guides/ai-crawler-access/ --- Diagnostic guide # Can AI crawlers read your site? Checking access agent by agent The AI crawlers and fetchers each platform documents, why training and retrieval access are separate decisions, how robots.txt matching works, why firewalls and JavaScript rendering block more sites than robots.txt does, and a verification sequence you can run in twenty minutes. Author Dorian Menard, founder of Search Scope Published 9 September 2026 Reading time 8 min ## The agents that matter, and what each does DOCUMENTED Every entry in this table is taken from the operator’s own documentation, read on 9 September 2026. Names change and new agents appear, so the operator page is the source to check, not this table. User agent Operator Documented purpose GPTBot OpenAI Crawls content that may be used in training foundation models OAI-SearchBot OpenAI Surfaces websites in ChatGPT’s search features ChatGPT-User OpenAI Fetches a page when a user asks a question; not automatic crawling ClaudeBot Anthropic Collects content that could contribute to model training Claude-SearchBot Anthropic Indexes content to improve search results within Claude Claude-User Anthropic Fetches a page when a Claude user asks a question PerplexityBot Perplexity Surfaces and links websites in Perplexity results; not for training Perplexity-User Perplexity Fetches a page for a user’s question and links to it; generally ignores robots.txt Google-Extended Google robots.txt token (no separate user agent) controlling Gemini training and grounding; no effect on Search Googlebot Google Search crawling, which also governs AI Overviews and AI Mode ## Training access and retrieval access are separate decisions DOCUMENTED Each operator separates the agent that gathers training data from the agent that serves answers, and lets a site owner control them independently. OpenAI: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. Anthropic: restricting ClaudeBot “signals that the site’s future materials should be excluded from our AI model training datasets”, while disabling Claude-SearchBot “prevents our system from indexing your content for search”. Google: Google-Extended “does not impact a site’s inclusion in Google Search”. The expensive mistake is blocking retrieval while meaning to block training. A blanket `Disallow: /` added in 2023 to stop model training also removes a site from the answers its buyers read today. Either restriction can be defended on purpose: a publisher with a licensing position has a case for blocking training; a Perth service business trying to be recommended has almost no case for blocking retrieval. The point is to make one decision without getting the other outcome by accident. ## Reading and writing robots.txt for these agents DOCUMENTED robots.txt directives are matched per user-agent group, and a named group takes precedence over the wildcard group for that agent. A `Disallow: /` under `User-agent: *` does not apply to an agent that has its own group, and an agent without its own group falls back to the wildcard. Most misconfigurations trace to that one rule. ``` # Allow retrieval, restrict training User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: * Allow: / ``` DOCUMENTED Two documented caveats. Perplexity-User “generally ignores robots.txt rules” because a user requested the fetch, and OpenAI says the same of ChatGPT-User. And OpenAI notes its search systems take about 24 hours to reflect a robots.txt change. A `noindex` meta tag or an `X-Robots-Tag` header operates at a different layer from robots.txt and removes a page from Google’s index regardless of what robots.txt allows; Google’s eligibility rule for AI features is that the page is indexed. ## Firewalls block more sites than robots.txt does DOCUMENTED Perplexity’s documentation says outright that a site behind a web application firewall “may need to explicitly whitelist Perplexity’s bots” and publishes IP ranges for the purpose. OpenAI publishes IP ranges for its agents for the same reason. Google’s guidance for AI features lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”. INFERRED Bot-management rules written to stop scrapers catch AI agents nobody meant to exclude, and the block is invisible from the browser and from robots.txt. When the raw HTML test and the server logs disagree with what robots.txt says should happen, the CDN or firewall is the usual cause. ## The rendering problem INFERRED Googlebot renders JavaScript. No other operator in the table documents that its retrieval agent executes JavaScript, and none should be assumed to. A site whose main content is assembled in the browser can deliver a near-empty document to an agent that fetches the raw HTML. The safe assumption is that the agent executes no JavaScript, then check whether the content survives that assumption. The test: open the page’s raw source (view-source, or `curl` the URL) and search for a distinctive sentence from the visible content. Present means a non-rendering agent can read it. Absent means it cannot, and the fix is server-side rendering or static generation for the pages that matter, which is engineering work and worth discovering early. ## A verification sequence you can run today 1. **Read robots.txt.** Open `/robots.txt`, list every named agent group and what each allows. Note which agents from the table have no group and therefore inherit the wildcard. 2. **Fetch the raw HTML.** `curl -A "OAI-SearchBot" https://yoursite.com.au/service-page/` (repeat with other agent names) and search the output for a sentence from the page. Compare with a normal browser fetch. A 403, a challenge page or an empty body is a firewall or rendering finding. 3. **Check headers and meta.** Look for `X-Robots-Tag` in the response headers and `` in the source. 4. **Check the CDN or firewall rules.** Bot-management and “AI scraper” toggles in Cloudflare and similar products can block the agents in the table. Read the rule list, not the marketing name of the toggle. 5. **Read the server logs.** Search for the agent names. Presence confirms they reach the server; absence over weeks, with robots.txt allowing them, points upstream. Twenty minutes covers a small site. Write the resulting policy down: which agents are allowed, which are restricted, and why. Review it when a platform publishes a new agent. ## What is documented and what is inferred - **Documented:** every agent’s purpose; the training-versus-retrieval independence; robots.txt precedence; Perplexity-User and ChatGPT-User behaviour; firewall guidance; the 24-hour propagation. - **Inferred:** that firewalls are the most common invisible block; that non-Google retrieval agents do not render JavaScript. - **Not established:** the volume of any agent’s traffic to Perth sites, which this publication has not measured. ## Related pages - [ Reference AI search optimisation Why retrieval access is the first layer. ](https://aiseoperth.net.au/ai-search-optimisation/) - [ Reference ChatGPT SEO OpenAI's agents in detail. ](https://aiseoperth.net.au/chatgpt-seo/) - [ Reference Perplexity SEO Perplexity's agents and firewall guidance. ](https://aiseoperth.net.au/perplexity-seo/) - [ Reference Gemini SEO What Google-Extended controls. ](https://aiseoperth.net.au/gemini-seo/) - [ Guide Why pages are not appearing in AI Overviews Access is cause one. ](https://aiseoperth.net.au/guides/not-appearing-in-ai-overviews/) ## Sources 1. [Overview of OpenAI crawlers](https://platform.openai.com/docs/bots), OpenAI (read 9 September 2026) DOCUMENTED 2. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic (read 9 September 2026) DOCUMENTED 3. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), Perplexity (read 9 September 2026) DOCUMENTED 4. [Google crawlers and fetchers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), Google Search Central (read 9 September 2026) DOCUMENTED 5. [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), Google Search Central (read 9 September 2026) DOCUMENTED On this page - [The agents that matter, and what each does](#the-agents-that-matter-and-what-each-does) - [Training access and retrieval access are separate decisions](#training-access-and-retrieval-access-are-separate-decisions) - [Reading and writing robots.txt for these agents](#reading-and-writing-robotstxt-for-these-agents) - [Firewalls block more sites than robots.txt does](#firewalls-block-more-sites-than-robotstxt-does) - [The rendering problem](#the-rendering-problem) - [A verification sequence you can run today](#a-verification-sequence-you-can-run-today) - [What is documented and what is inferred](#what-is-documented-and-what-is-inferred) - [Sources](#sources) Evidence labels Claims on this page are labelled DOCUMENTED, OBSERVED, TESTED, INFERRED or HYPOTHESIS. [What the labels mean](https://aiseoperth.net.au/methodology/#evidence-labels). Published by Search Scope AI SEO Perth is published by Search Scope, a Perth SEO consultancy founded by Dorian Menard. Search Scope provides commercial SEO and AI search optimisation services; this website is maintained as its specialist Perth AI search research and reference publication. [Search Scope](https://searchscope.com.au/)