Skip to main content

Reference

AI search optimisation: how AI-assisted search retrieves and cites sources

Author
Dorian Menard, founder of Search Scope
Published
Last updated
Reading time
8 min

AI search optimisation is the work of making a source usable by AI-assisted search: systems that retrieve documents at answer time, compose a response with a language model, and attach citations to some of what they used. It differs from optimising for a ranked list because the unit is the retrieved passage and the outcome is being composed from and cited, not a position. What the platforms document is access and eligibility; how they select among eligible sources is largely undisclosed.

Key facts

Key facts about AI search optimisation
Fact Detail Evidence
Two knowledge paths A model can answer from what it learned in training or from documents retrieved at answer time. Only retrieval can be influenced by a site owner today; platforms document separate crawlers for each. DOCUMENTED
Query fan-out Google documents that AI Overviews and AI Mode may issue multiple related searches across subtopics and data sources to develop a response. DOCUMENTED
Citations Perplexity documents that its user-triggered fetcher includes a link to the page in its response; Google documents supporting links in AI Overviews and AI Mode. Citation display and completeness differ by platform. DOCUMENTED
Retrieval providers Which web index each assistant searches is not fully documented by every platform. This site does not assert an undisclosed provider. DOCUMENTED
Variance Measured. Of the businesses recommended for a Perth commercial question, only 8% to 31% depending on the platform were recommended in all three runs of the same question. OBSERVED

What AI-assisted search is

AI-assisted search is any search experience where a generative model composes the answer. It includes AI Overviews and AI Mode inside Google Search, chat assistants such as ChatGPT, Gemini, Claude and Copilot when they search the web, and citation-first engines such as Perplexity. What they share is a pipeline with three visible steps: retrieve documents, compose an answer, attribute some of the sources used. What differs is how much of each step the platform documents.

Training knowledge versus retrieval

DOCUMENTED Every major platform separates the crawler that gathers training data from the agent that fetches pages to answer a question, and lets site owners control them separately. OpenAI documents GPTBot for training and OAI-SearchBot for its search features, and states that each robots.txt setting “is independent of the others”. Anthropic documents ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-initiated fetches. Perplexity documents PerplexityBot for its search index and Perplexity-User for user actions, and says neither is used to collect content for training foundation models. Google’s Google-Extended token controls training and grounding for Gemini apps and “does not impact a site’s inclusion in Google Search”.

The consequence for optimisation is that what a model learned in training cannot be edited, and what it retrieves can be. AI search optimisation is retrieval-side work.

How retrieval happens

DOCUMENTED Google is the platform that documents the most. Its site-owner guidance says AI Overviews and AI Mode “may use a query fan-out technique, issuing multiple related searches across subtopics and data sources, to develop a response”, and that while responses are generated its models “identify more supporting web pages”, showing “a wider and more diverse set of helpful links” than a classic search. Google’s AI Mode announcement describes the same technique as “breaking down your question into subtopics and issuing a multitude of queries simultaneously”. Eligibility is the normal Search index: a page must be indexed and eligible for a snippet.

DOCUMENTED Gemini’s privacy documentation states that “some Gemini responses are grounded on Search results”, which places the Gemini app on the same retrieval substrate for grounded answers.

DOCUMENTED OpenAI documents that OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features” and that ChatGPT-User “may visit a web page” when a user asks a question, but is “not used for crawling the web in an automatic fashion”. Perplexity documents the equivalent pair. Neither publishes how candidate pages are ranked once retrieved.

What is not documented, by any platform, is the selection step: given several eligible, retrieved pages, which ones the answer draws on and why. Every ranked list of “AI ranking factors” in circulation is inference or observation dressed as documentation.

How citations work

DOCUMENTED Perplexity documents that its user-triggered fetcher “might visit a web page to help provide an accurate answer and include a link to the page in its response”. Google documents that AI Overviews and AI Mode show links to supporting websites and describes AI Mode as returning “prominent links for people to click on”. OpenAI’s documentation frames OAI-SearchBot opt-out as removal from “ChatGPT search answers” while sites “can still appear as navigational links”.

INFERRED Two inferences follow and are labelled as such. First, a source can inform an answer without being cited, because no platform claims every retrieved document is attributed; counting visible citations therefore undercounts influence. Second, citation display varies by surface, so absence of a citation in one run is not evidence that the source was unused. This publication’s methodology records citations as shown and treats absence as a data point, not a conclusion.

Why the same prompt gets different answers

OBSERVED Repeated runs of the same prompt return different sources and different brands, and the size of that effect is now measured rather than asserted. In the Perth AI Search Study 2026, 200 commercial Perth questions were asked three times over three days on four platforms. The share of recommended businesses that survived all three runs was 31% on Google AI Mode, 17% on ChatGPT, 11% on Google AI Overviews and 8% on Gemini. Whether an AI Overview appeared at all was unstable for roughly a third of questions: 91 of 200 produced one in all three rounds, 71 in some rounds only, and 38 never. HYPOTHESIS The contributors remain untested individually: the model’s sampling, the searches it constructs, personalisation, inferred location, logged-in state and ongoing model changes. Platforms document some of these separately and none publishes the combined effect, so this site measures the result and reports divergence.

What AI search optimisation is, given all that

Take the documented parts and the work follows:

  1. Be retrievable. Allow the search and user agents named above through robots.txt and any firewall or CDN rule, serve content as HTML, and stay indexed on Google.
  2. Be the answer to the sub-questions. Fan-out means a page can be retrieved for a sub-query the person never typed. Coverage of the comparison, cost, suitability and local questions around a subject is what gives an engine something to retrieve.
  3. Be identifiable. An answer that names a business has to resolve it to one entity across the sources it retrieved. Entity SEO is that work.
  4. Be corroborated. Retrieval returns several sources; the ones that agree get composed from. Independent sources stating the same facts matter because the engine reads them too.
  5. Measure with repetition. One run is an anecdote.

How it applies in Perth

OBSERVED A Perth prompt is a local prompt, and the sources composed into Perth answers have now been counted rather than assumed. The first observation round found the mix differs by platform more than it differs by question: Google AI Mode cited Google-hosted business panels for 85.8% of its citations, AI Overviews cited businesses’ own websites for 38.2% to 70.9% of its citations and Gemini for 42.0% to 77.6% (the lower figure counts only businesses the study identified, the higher every site that looks like a business website), and ChatGPT never cited Google across 1,794 citations. Between 60% and 78% of the businesses recommended did not appear in the Google Maps top 20 for the same question, so the local pack is not what these systems reheat. The documented Perth claim on this page still holds: both Google surfaces are live in Australia, and the retrieval agents above are the ones a Perth site has to let in.

What is documented and what is inferred

  • Documented: the training and retrieval split per platform; Google’s fan-out and eligibility rule; Gemini grounding on Search; Perplexity’s citation behaviour; OpenAI’s opt-out consequence.
  • Inferred: uncited influence; citation display varying by surface.
  • Observed: the size of run-to-run variance, and the Perth source mix per platform, in the Perth AI Search Study 2026.
  • Hypothesis: which mechanisms produce that variance. The effect is measured; its causes are not.
  • Undisclosed: each platform’s selection logic among retrieved pages, and any web index provider a platform has not named.

Sources

  1. AI features and your website, Google Search Central (read 9 September 2026) DOCUMENTED
  2. AI Mode in Search gets new agentic features and expands globally, Google Blog, 20 May 2025 (read 9 September 2026) DOCUMENTED
  3. Overview of OpenAI crawlers, OpenAI (read 9 September 2026) DOCUMENTED
  4. Perplexity crawlers, Perplexity (read 9 September 2026) DOCUMENTED
  5. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic (read 9 September 2026) DOCUMENTED
  6. Google crawlers and fetchers (Google-Extended), Google Search Central (read 9 September 2026) DOCUMENTED
  7. Gemini Apps Privacy Hub, Google (read 9 September 2026) DOCUMENTED

    States that some Gemini responses are grounded on Search results.