Allow the AI search crawlers if you want those engines to find your pages: OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are a separate decision. llms.txt is optional, and Google says its Search does not use the file.
By Donnie Strompf. Updated October 2026.
Key takeaways
- Four companies (OpenAI, Anthropic, Google, Apple) document separate crawlers for search and for model training, so one rule for “AI bots” is too blunt.
- OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce visibility and accuracy in Claude search results.
- Google says Google-Extended does not affect inclusion in Google Search. Apple says pages that disallow Applebot-Extended can still be included in search results.
- A robots.txt allow does nothing if your firewall or CDN refuses the bot, and some user-requested fetchers ignore the file.
- Google’s guidance (updated July 2026) says you do not need llms.txt to appear in Google Search, and that creating one is fine.
Which AI crawlers should you allow in robots.txt?
I’m Donnie Strompf, and I’ve done SEO since 2008. The table uses only what each company publishes about its own crawlers, all read on October 15, 2026. The last column gives the page and the date it shows.
The documentation separates three jobs. Search crawlers help an engine find pages that may be named in an answer. Training crawlers collect content that may be reused to build models. Fetchers load a page because a person asked.
How each engine uses its crawler and picks among the pages it can read is covered in our guide to how AI search engines choose their sources. The table below covers only what robots.txt controls.
| Job | robots.txt token | What blocking it stops, per the company | Does it respect robots.txt? | Page read |
|---|---|---|---|---|
| Search | Googlebot | Crawling for Google Search, which AI Overviews and AI Mode draw on. | Yes: Google says its crawlers “always obey robots.txt rules when crawling automatically.” | Google crawler list, October 15, 2026; AI features, December 10, 2025 |
| Search | OAI-SearchBot | Your site is left out of ChatGPT search answers, apart from plain navigational links. | Yes: robots.txt is the documented opt-out. | OpenAI bots, no date shown |
| Search | PerplexityBot | The page states no effect. It says the bot is for surfacing sites in Perplexity results and recommends allowing it. | Yes: Perplexity documents robots.txt for it. | Perplexity bots, no date shown |
| Search | Claude-SearchBot | Indexing for search quality, so visibility and accuracy in user search results may fall. | Yes: Anthropic says its bots honor robots.txt directives. | Anthropic article, April 7, 2026 |
| Search | bingbot | The post does not say. It explains that a bingbot group replaces every other group in the file. | Yes: the post describes how the bot reads the file. | Bing Webmaster Blog, May 3, 2012 |
| Search and training | Applebot | Apple’s crawl for Siri and Safari search features, which may also feed model training. | Yes, and it follows Googlebot rules if it has none of its own. | Apple page, September 4, 2026 |
| Training | GPTBot | Use of your content to train OpenAI’s models. | Yes: disallowing it is the documented signal, set independently of OAI-SearchBot. | OpenAI bots, no date shown |
| Training | ClaudeBot | Use of your site’s future materials in Anthropic’s training datasets. | Yes: Anthropic says its bots honor robots.txt directives. | Anthropic article, April 7, 2026 |
| Training | Google-Extended | Gemini training and grounding in Gemini Apps. Google says it “does not impact a site’s inclusion in Google Search.” | It is a robots.txt token with no user agent of its own, applied by Google’s crawlers. | Google crawler list, October 15, 2026 |
| Training | Applebot-Extended | Only how Applebot’s data is used. Pages “can still be included in search results.” | Yes: a separate robots.txt user agent that never crawls. | Apple page, September 4, 2026 |
| Training | Amazonbot | Amazon’s crawl, whose data may train Amazon AI models. A noarchive meta tag opts a page out. | Yes: Amazon says it honors the exclusion protocol. | Amazonbot page, no date shown |
| Training | meta-externalagent | Meta’s crawl for training models or indexing content for its products. | Not stated. The page shows a robots.txt example. | Meta crawler page, no date shown |
| Archive | CCBot | Common Crawl’s open archive of web crawl data. The page does not mention AI training. | The page gives a block rule and warns of crawlers falsely claiming to be CCBot. | CCBot page, no date shown |
| User-requested fetch | ChatGPT-User | Page loads when a ChatGPT user asks. Not used to decide Search appearance. | Not reliably: “robots.txt rules may not apply.” | OpenAI bots, no date shown |
| User-requested fetch | Perplexity-User | Page loads when a Perplexity user asks. | No: the fetcher “generally ignores robots.txt rules.” | Perplexity bots, no date shown |
| User-requested fetch | Claude-User | Retrieval for a user’s question, so visibility in user-directed search may fall. | Yes: Anthropic says its bots honor robots.txt directives. | Anthropic article, April 7, 2026 |
| User-requested fetch | meta-externalfetcher | Page loads when a Meta user asks. | No: it “may bypass robots.txt.” | Meta crawler page, no date shown |
If you want visibility in AI search, check that the search crawlers in the table can reach your public pages. Blocking a training bot is a choice about your content, and the effect depends on the bot and the product. Each of the four companies in the first takeaway describes its training controls as separate from search.
How do you write the robots.txt rules?
A robots.txt file is a list of groups, and each group opens with a User-agent line. Google’s specification page (last updated August 31, 2026) says only one group is valid for a crawler: the one with the most specific matching user agent. Specific groups and the * group are not combined.
A named group takes precedence over the wildcard group, which has two effects. A bot you name in its own group ignores your * group, so repeat any rules it needs. Anthropic says the file belongs in the top-level directory of each host and must be repeated for each subdomain.
| Goal | Groups to add |
|---|---|
| Stay in search and opt out of training at four companies | One group each for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, each with Disallow: /. Leave the search bots (OAI-SearchBot, Claude-SearchBot, Googlebot, Applebot) out of the file. |
Robots.txt is also the wrong tool for hiding a private page. Google’s introduction (last updated December 10, 2025) calls it “not a mechanism for keeping a web page out of Google” and points to noindex or a password. Our guide to what technical SEO covers shows how crawling and indexing relate to robots.txt.
What can block an AI crawler even when robots.txt allows it?
Your host, CDN or security plugin can refuse a bot before it reads the file. Three vendors document how network-level blocking affects their crawlers:
- OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing its published IP ranges.
- Perplexity says that behind a web application firewall “you may need to explicitly whitelist Perplexity’s bots.” Its examples combine a user-agent match with a match against its published IP lists.
- Anthropic says blocking IP addresses “may not work correctly or persistently guarantee an opt-out,” because it can stop Anthropic from reading robots.txt at all.
Check your CDN’s bot settings separately from robots.txt. Cloudflare’s blog post of July 1, 2026 describes three categories available to every customer (Search, Agent, Training). From September 15, 2026, it says, new domains block Training and Agent crawlers by default on pages that show ads, while Search stays allowed.
It also names Googlebot and Applebot, along with BingBot, as multi-purpose crawlers that customers will block if they choose to block Training. That date has passed, so check the bot setting in your own account.
To see who arrived, read your server logs or CDN bot report. A user-agent string can be copied by anyone, so match the IP address to the company’s list: OpenAI publishes files such as searchbot.json, Perplexity perplexitybot.json and Anthropic claude.com/crawling/bots.json.
Does llms.txt do anything?
llms.txt is a proposal for a Markdown file that lists the pages an AI tool should read first. The sources below show what each party says about it.
| Source | What it says | Date |
|---|---|---|
| llmstxt.org | Jeremy Howard’s proposal, an informal overview open for community input. Thousands of sites publish the file, and several AI labs publish one for their developer docs. | Published September 3, 2024; modified August 10, 2026 |
| Google, AI optimization guide | “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search,” because “Google Search itself doesn’t use them.” Creating one is “completely fine.” | Last updated July 10, 2026 |
| Google, AI features | “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” | Last updated December 10, 2025 |
| OpenAI bots and Perplexity bots | Each links an llms.txt index of the company’s own documentation. Neither page says its crawlers read the file on other sites. | No date shown |
| Anthropic article | Does not mention llms.txt. | April 7, 2026 |
The crawler pages I read from OpenAI and Perplexity, and the Anthropic article, describe robots.txt and IP lists, and none says it reads llms.txt from your site. No source above promises a result from it.
If you publish one, list only pages that exist and update it when your prices change. Be wary of anyone who sells llms.txt as a ranking or citation tool, and ask which company documents reading it. Other steps that may affect whether your pages are included are in our guide on how to get recommended by ChatGPT and Google AI Overviews.
What should you ask your host or developer about bots?
- Can I see the live
/robots.txtfor every host and subdomain, and which groups name AI bots? - Does the CDN, host or security plugin block bots by default, and what is each AI bot setting today?
- Which log lines, with date, user agent and IP address, show that the search bots reached the site?
- If llms.txt is offered as a paid item, which company documents using it?
Our AEO services page describes the work, and the SEO pillar guide covers the search basics the AI features rest on. AEO costs $950 to $2,000 a month alongside SEO, and every service price is on the pricing page. Our guide to how much AEO costs sets out the ranges.
FAQ
Will blocking Google-Extended remove me from AI Overviews?
Google documents no such link. It says Google-Extended does not impact inclusion in Google Search, and its AI features page names Googlebot rules for access and nosnippet, data-nosnippet, max-snippet and noindex for what is shown. Google-Extended governs Gemini training and Gemini Apps grounding, which are different products.
If I block a bot in robots.txt, can an AI still read my page?
Sometimes. OpenAI says ChatGPT-User fetches may not follow robots.txt, Perplexity says Perplexity-User generally ignores it, and Meta says meta-externalfetcher may bypass it. These bots act when a person asks about your page. Anthropic says its bots honor robots.txt. To stop a fetch entirely, use a login or a firewall rule.
How long does a robots.txt change take to apply?
Expect a delay. Meta asks you to allow up to 24 hours and says crawlers may cache robots.txt for up to 24 hours, and OpenAI gives about 24 hours for search results. Microsoft’s 2012 post says Bing keeps your directives in memory for a few hours.
Should I set Crawl-delay for AI bots?
Support differs. Anthropic says it supports the non-standard Crawl-delay extension where appropriate. Amazon and Apple say Amazonbot and Applebot do not follow it, and the OpenAI and Perplexity pages I read do not mention it. If a bot strains your server, ask your host to rate-limit it by IP address.
What goes in an llms.txt file and where does it live?
Per llmstxt.org, it is a Markdown file at the site root or a path within it. It needs an H1 naming the project, and can add a short blockquote summary and H2 sections that list links to your key pages. It is a proposal, so none of the vendors above promises to read it.
Related reading
- What is answer engine optimization (AEO)?
- How to fix wrong information about your business in ChatGPT and Google AI Overviews
- Schema markup for a small business: which types to add first
Next step
The free AI-visibility audit comes first, then a 15-minute call.
Get your free AI-visibility audit
Marketing that works, in your inbox
AI search & SEO news, what’s working right now, and guides we actually use with clients. A couple of emails a month, no spam.
Unsubscribe anytime. We treat your inbox the way we treat our clients’ brands.