Can AI crawlers read your site? Checking access agent by agent

The agents that matter, and what each does

DOCUMENTED Every entry in this table is taken from the operator’s own documentation, read on 9 September 2026. Names change and new agents appear, so the operator page is the source to check, not this table.

Training access and retrieval access are separate decisions

DOCUMENTED Each operator separates the agent that gathers training data from the agent that serves answers, and lets a site owner control them independently. OpenAI: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. Anthropic: restricting ClaudeBot “signals that the site’s future materials should be excluded from our AI model training datasets”, while disabling Claude-SearchBot “prevents our system from indexing your content for search”. Google: Google-Extended “does not impact a site’s inclusion in Google Search”.

The expensive mistake is blocking retrieval while meaning to block training. A blanket Disallow: / added in 2023 to stop model training also removes a site from the answers its buyers read today. Either restriction can be defended on purpose: a publisher with a licensing position has a case for blocking training; a Perth service business trying to be recommended has almost no case for blocking retrieval. The point is to make one decision without getting the other outcome by accident.

Reading and writing robots.txt for these agents

DOCUMENTED robots.txt directives are matched per user-agent group, and a named group takes precedence over the wildcard group for that agent. A Disallow: / under User-agent: * does not apply to an agent that has its own group, and an agent without its own group falls back to the wildcard. Most misconfigurations trace to that one rule.

# Allow retrieval, restrict training
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

DOCUMENTED Two documented caveats. Perplexity-User “generally ignores robots.txt rules” because a user requested the fetch, and OpenAI says the same of ChatGPT-User. And OpenAI notes its search systems take about 24 hours to reflect a robots.txt change. A noindex meta tag or an X-Robots-Tag header operates at a different layer from robots.txt and removes a page from Google’s index regardless of what robots.txt allows; Google’s eligibility rule for AI features is that the page is indexed.

Firewalls block more sites than robots.txt does

DOCUMENTED Perplexity’s documentation says outright that a site behind a web application firewall “may need to explicitly whitelist Perplexity’s bots” and publishes IP ranges for the purpose. OpenAI publishes IP ranges for its agents for the same reason. Google’s guidance for AI features lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”.

INFERRED Bot-management rules written to stop scrapers catch AI agents nobody meant to exclude, and the block is invisible from the browser and from robots.txt. When the raw HTML test and the server logs disagree with what robots.txt says should happen, the CDN or firewall is the usual cause.

The rendering problem

INFERRED Googlebot renders JavaScript. No other operator in the table documents that its retrieval agent executes JavaScript, and none should be assumed to. A site whose main content is assembled in the browser can deliver a near-empty document to an agent that fetches the raw HTML. The safe assumption is that the agent executes no JavaScript, then check whether the content survives that assumption.

The test: open the page’s raw source (view-source, or curl the URL) and search for a distinctive sentence from the visible content. Present means a non-rendering agent can read it. Absent means it cannot, and the fix is server-side rendering or static generation for the pages that matter, which is engineering work and worth discovering early.

A verification sequence you can run today

1. Read robots.txt. Open /robots.txt, list every named agent group and what each allows. Note which agents from the table have no group and therefore inherit the wildcard.
2. Fetch the raw HTML. curl -A "OAI-SearchBot" https://yoursite.com.au/service-page/ (repeat with other agent names) and search the output for a sentence from the page. Compare with a normal browser fetch. A 403, a challenge page or an empty body is a firewall or rendering finding.
3. Check headers and meta. Look for X-Robots-Tag in the response headers and <meta name="robots"> in the source.
4. Check the CDN or firewall rules. Bot-management and “AI scraper” toggles in Cloudflare and similar products can block the agents in the table. Read the rule list, not the marketing name of the toggle.
5. Read the server logs. Search for the agent names. Presence confirms they reach the server; absence over weeks, with robots.txt allowing them, points upstream.

Twenty minutes covers a small site. Write the resulting policy down: which agents are allowed, which are restricted, and why. Review it when a platform publishes a new agent.

What is documented and what is inferred

• Documented: every agent’s purpose; the training-versus-retrieval independence; robots.txt precedence; Perplexity-User and ChatGPT-User behaviour; firewall guidance; the 24-hour propagation.
• Inferred: that firewalls are the most common invisible block; that non-Google retrieval agents do not render JavaScript.
• Not established: the volume of any agent’s traffic to Perth sites, which this publication has not measured.