Jake Young
Most brands that never appear in AI answers assume their content isn’t good enough. Usually the content was never read. A firewall turned the crawler away, or the page only existed after JavaScript ran, or the answer to the buyer’s question was buried in paragraph nine where no retrieval system would find it.
When ChatGPT, Perplexity, Claude or Google’s AI Mode cites three sources, those pages survived a pipeline with five gates, and the first two are pure engineering. Fix them and your content finally gets judged. Then you can build pages that win the last three, passage by passage.
How Answer Engines Choose Sources
Most AI search products are retrieval augmented generation systems. The model rewrites the question into one or more searches, pulls candidate pages from an index, splits them into passages, ranks the passages, and writes an answer grounded in the few that survive, citing some along the way.
Google says AI Overviews and AI Mode may use a “query fan-out” technique, issuing multiple related searches across subtopics to build a response. Fan-out changes who gets cited. Ahrefs looked at 863,000 keywords and found only 38% of pages cited in AI Overviews also ranked in the top ten for the original query, down from 76% in its July 2025 study, though it notes its detection improved in between. The direction is what matters: you compete for the sub-questions as well as the head term.
Being listed and being used are separate wins. A 2026 study by Zhang, He and Yao analysed 21,143 citations across three AI search platforms and found the pages with most influence on answers tended to be long and well structured, with extractable evidence such as definitions, numbers, comparisons and steps.
The citation is also becoming the whole impression. Pew found people clicked a traditional result in 8% of visits when a Google AI summary appeared, against 15% without one, and clicked a link inside the summary in 1% of visits. Many buyers will meet your brand as one sentence in someone else’s answer.
The Five Gates
We audit every site against five gates, in order, because failing one makes the rest irrelevant.
1. Access: is the bot allowed in?
robots.txt, CDN rules, firewalls and rate limits all get a vote. This is the most common failure we find and the easiest to fix, so it gets its own walkthrough below.
2. Render: can it read the page without JavaScript?
Vercel’s analysis of AI crawler traffic found that of the major AI crawlers, only Google’s Gemini and Apple’s rendered JavaScript. OpenAI’s, Anthropic’s and Perplexity’s fetched script files without running them. If your offer, specifications or article body appear only after hydration, they don’t exist for most answer engines.
3. Index: is the page where the assistant searches?
Google’s AI features draw on its Search index, and Google says there are no extra technical requirements beyond being indexed and eligible for a snippet. Other assistants run their own search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block one and you leave that assistant’s index, whatever your content says.
4. Retrieve: does a passage match the sub-question?
Retrieval works on passages. A page that answers twelve questions in one flowing essay may match none of them strongly. A page with a clear section for each, opening with a direct answer, can match all twelve.
5. Quote: is the passage worth citing?
The model uses passages that answer precisely and carry their own evidence. The original GEO paper by Aggarwal and colleagues found that adding citations, quotations and statistics could lift a source’s visibility in generative answers by up to 40%, varying by domain.
Walkthrough: Writing a Crawler Policy
Most brands have never decided which AI bots they allow. A security team or a CDN default decided for them.
Sort the bots into three groups, because each needs its own decision. Training crawlers collect content that may train future models: GPTBot and ClaudeBot, plus Google-Extended, a control token that covers Gemini training and grounding. Search crawlers build the indexes assistants cite from: OAI-SearchBot, Claude-SearchBot and PerplexityBot. User triggered fetchers load a page because a person asked about it: ChatGPT-User, Claude-User and Perplexity-User.
Read the fine print, because the groups behave differently. OpenAI says robots.txt changes take about 24 hours to reach OAI-SearchBot, and that its rules may not apply to ChatGPT-User because a person started the fetch. Perplexity says Perplexity-User generally ignores robots.txt for the same reason, while Anthropic says all three of its bots respect it. Google-Extended doesn’t affect Google Search. To limit how pages appear in AI Overviews, you use nosnippet, data-nosnippet, max-snippet or noindex.
Decide each group on purpose. For most brands we recommend allowing search crawlers and user fetchers without hesitation, since blocking them removes you from answers. Training crawlers are the real decision. A publisher that sells its content has good reasons to block them. A brand that wants to be known usually doesn’t, because training data is where a model’s default impression of you comes from. Write the decision down and name an owner.
Give each bot its own User-agent group with an explicit rule, so a change to the wildcard group can’t silently rewrite your AI policy. Keep the file fast and always available: Google stops crawling for up to 12 hours when robots.txt returns a server error, so a failed deploy can become a small search outage.
Then check the layers robots.txt doesn’t control. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains unless the owner opts in, and many security teams challenge any unfamiliar bot. Fetch key pages with each bot’s user agent, look for 403s and challenge pages, and read server logs by user agent and status. User agents are easy to spoof, so verify against the IP ranges OpenAI, Perplexity and others publish before you allowlist anything.
Walkthrough: Engineering a Page for Retrieval
Take a service page that should be cited when buyers research the category. We rebuild it in five steps.
First, write the fan-out yourself. List the sub-questions a buyer asks on the way to a decision: what it is, who it suits, how it works, how pricing is structured, how long it takes, what can go wrong, how it compares. Pull candidates from sales calls, site search logs and the follow up questions assistants suggest, then cut to the dozen that matter. The cut is a strategy decision, because it chooses which conversations you want to be part of.
Second, give each sub-question its own section. The heading states the question in the buyer’s words. The first sentence answers it in a form that survives being lifted out of the page: name the subject instead of writing “it”, include the number and the unit, and say who the answer applies to.
Third, put evidence directly after the claim: a statistic with its source, a quotation with its speaker, a date on anything that changes.
Fourth, keep facts in text. Tables in HTML, results written out rather than baked into images, alt text that describes content. Then load the page with JavaScript disabled and read what’s left.
Fifth, sign and date it, with an author who has real expertise.
Two trade-offs. Writing for extraction can flatten a brand voice into FAQ filler, so we keep the first sentence of each section plain and let the second carry the voice. And don’t shred the page: Google’s own guidance says you don’t need to break content into small chunks for AI. The goal is sections a person enjoys that also survive being quoted alone.
Entities, Schema and llms.txt
An answer engine has to know which company a passage is about. One canonical name, Organization markup with a stable @id and sameAs links, and matching descriptions across your profiles do that job. Google says structured data isn’t required for its generative features, and we use it for disambiguation rather than ranking. The full method is in Your Brand Is a Dataset Now.
llms.txt, proposed by Jeremy Howard in 2024, is a Markdown summary at the site root with links to clean versions of key pages. It helps in developer documentation, where coding agents fetch it. For marketing sites the evidence is thin: Google says Search ignores it, and SE Ranking found no measurable effect on citations across 300,000 domains. Generate one if it’s cheap and don’t count it as progress.
Measuring AI Visibility Without Fooling Yourself
Answers vary across runs, prompts and time, which is why Schulte, Bleeker and Kaufmann argue that AI visibility has to be measured as a distribution from repeated runs. A screenshot of one good answer proves nothing.
Build the panel from your fan-out list. Run each sub-question many times on each assistant in a clean, logged out session, and record whether you were cited, which URL, and which passage the answer drew on. That last one tells you whether you shaped the answer or only decorated it.
Server logs add a signal most teams ignore. User triggered fetches from ChatGPT-User, Claude-User and Perplexity-User show pages being pulled into live answers because someone asked. Grouped by path each week, they form a rough map of which pages answer engines reach for. Combine them with referrals tagged utm_source=chatgpt.com and Google’s new AI reports in Search Console, and report ranges, never ranks.
What Changes Next
Most of this audit is mechanical and should be scripted to run monthly. The decisions take judgement: crawler policy, which questions matter commercially, and passages that stay in the brand’s voice. Handed to a model unsupervised, pages tend to come back sounding like the same encyclopedia entry, and a passage gets cited for saying something specific.
Our prediction: within two years, crawler policy and passage structure become standing items in every release, reviewed like page speed, because a silent block is too expensive to discover months later.
What to Do This Quarter
Write the crawler policy, bot by bot, with an owner. Fetch your ten most important pages as each bot and with JavaScript disabled, and fix anything blocked or missing.
Rebuild your three most valuable pages around their sub-questions, with a plain first sentence, evidence next to each claim, an author and a date.
Start the panel: twenty sub-questions, ten runs each, on the four assistants your buyers use. Add user triggered fetches from your logs, and compare month to month.
Answer engines reward sites that are easy to reach, easy to read and worth quoting, and most of that work fits in a quarter. If you want to know which gate your site fails today, we’d be glad to run the audit with you and talk through what we find.
