robots.txt for AI Crawlers: What to Allow and Block in 2026

Why can't you treat AI crawlers as one thing?

Because the same company often runs three bots with three different jobs, and robots.txt lets you decide each one separately. Sorting bots by job makes every later decision easier.

• Training crawlers collect content that may be used to train future models. GPTBot and ClaudeBot sit here.

• Search crawlers build the index an AI search feature draws on. OAI-SearchBot, Claude-SearchBot and PerplexityBot sit here.

• User-triggered fetchers load a page because a person asked a question that needed it. ChatGPT-User, Claude-User and Perplexity-User sit here.

Training is a policy question about how your content gets reused. Search and user fetches are a visibility question: they decide whether an assistant can find and read you when someone asks about your category. Mix the two up and you end up opting out of the wrong thing.

Which AI crawlers should you know in 2026?

These are the bots most sites will meet in their logs. The details come from each company's published documentation, checked in October 2026. They change, so recheck before you ship.

Opts you out of that use. Google says it does not affect Search inclusion or ranking Yes Two small notes. Anthropic's current documentation lists three bots, so older names like anthropic-ai and Claude-Web, which still turn up in copied block lists, are not something to rely on. And OpenAI also runs OAI-AdsBot, which only checks pages submitted as ads in ChatGPT. You can ignore it unless you advertise there. Should you block AI crawlers or allow them? For most businesses that want customers to find them, allow the search and user-fetch bots. Whether to also allow training is a separate choice, and it is yours. It is not an SEO requirement either way. Allowing a bot makes you eligible. It does not guarantee a citation. OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search, and Perplexity says the same about PerplexityBot, but neither promises you a mention. Google is just as direct: meeting every requirement does not guarantee it will crawl, index or serve a page. Here is a simple way to sort it by type of site: • You sell something or want enquiries (a clinic, a D2C brand, a B2B service firm, an agency): allow search and user-fetch bots. Decide on training based on how you feel about your content being reused. • You sell the content itself (paid courses, premium research, a subscriber library): block training bots, and keep paid material behind a login rather than relying on robots.txt. • You run staging sites or private areas: robots.txt alone is not enough. Put them behind a login or an IP allowlist. One point is worth stating plainly. robots.txt is a request, not a lock. Well-behaved crawlers follow it. Anything that does not will ignore it, and user-triggered fetchers may not apply it at all.

What should your robots.txt look like?

Here is a starting point for the common case: visible in AI answers, opted out of training. Swap the private paths for your own.

# AI search and answer bots: allowed, minus private areas User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Disallow: /admin/ Disallow: /checkout/ Disallow: /cart/ # Training: opted out User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / # Everyone else, including Googlebot User-agent: * Disallow: /admin/ Disallow: /checkout/ Disallow: /cart/ Sitemap: https://www.example.com/sitemap.xml

The repeated Disallow lines are not a typo. A crawler follows only the most specific group that names it, so a bot with its own group ignores whatever sits under User-agent: *. Give a bot its own rules and you have to repeat your private paths there. Leave Googlebot alone. Google says robots.txt rules for Googlebot are how you manage crawling for Search, and AI Overviews and AI Mode are part of Search. To limit what Google shows from a page, it points to nosnippet, data-nosnippet, max-snippet and noindex instead. Google-Extended is the token for limiting AI training and grounding in some of Google's other systems. On Shopify, the robots.txt.liquid template is where this goes. On WordPress, your SEO plugin or a physical file in the site root does the job. Subdomains need their own file. What if robots.txt says yes but the bot still can't get in? Then the block is almost certainly upstream, in your CDN, firewall or hosting. robots.txt can say yes all day while a security rule returns a 403 before the bot ever reads it. Google's own guidance on AI features says crawling has to be allowed in robots.txt and by any CDN or hosting infrastructure. Check Cloudflare first if your site sits behind it. It began blocking known AI crawlers by default for new domains in 2025, and it has announced new defaults for new domains from 15 September 2026: search crawlers allowed, agent and training traffic blocked by default on pages that show ads. Help Net Security's summary of the change also notes that blocking training crawlers can catch multi-purpose crawlers such as Googlebot, Applebot and Bingbot, even when search is allowed. Open your bot and AI crawler settings and read what is actually switched on instead of assuming. Blocking by IP address can backfire too. Anthropic notes that IP-based blocking can stop its crawler from reading your robots.txt at all, which defeats the purpose. A quick test from your terminal: run curl -I -A "OAI-SearchBot" followed by your homepage URL. A 200 means no user-agent rule is catching that bot. A 403 or a challenge page means something is. This will not catch rules based on IP reputation, so check your logs as well.

How do you check it's working?

1. Open yoursite.com/robots.txt and read it as a stranger would. Look for a blanket Disallow: / under User-agent: * and for block lists someone pasted in without reading. 2. Search your server or CDN logs for each bot name separately. Count OAI-SearchBot, GPTBot and ChatGPT-User on their own, because they are different bots with different jobs. 3. Verify IP addresses, not just names. OpenAI, Perplexity and Anthropic publish IP lists for their crawlers, and Google documents a reverse DNS check. A user-agent string alone can be faked. 4. Give it time. OpenAI has said search-related robots.txt changes can take around a day to apply, and Google says recrawling can take anywhere from several days to several months depending on the page. 5. Watch referral traffic from AI assistants in GA4 over the next month. Our guide to tracking ChatGPT, Perplexity and Gemini traffic in GA4 shows how. Access only gets you eligible. Getting cited still comes down to the content, which we cover in how to get your brand cited in ChatGPT and AI search answers. If you are planning a rebuild, the enterprise website architecture guide covers rendering and crawlability.

Mistakes that quietly cost AI visibility

• Pasting a "block all AI" list from a blog without reading it. Many of them block the search bots you want.

• Assuming that blocking GPTBot hides you from ChatGPT search. OpenAI treats its bot settings as independent, and OAI-SearchBot controls search.

• Giving a bot its own group and forgetting to repeat your private paths inside it.

• Blocking IP ranges instead of using robots.txt.

• Forgetting subdomains. Each host serves its own robots.txt.

robots.txt decides who may come in. llms.txt is an optional pointer to your best pages, and no major AI lab has confirmed it reads the file. They do different jobs, so do not swap one for the other. Want a second opinion on your setup? Talk to our SEO, AEO and AI team.

Frequently Asked Questions

Will blocking GPTBot remove my site from ChatGPT?

No. OpenAI treats its bot settings as independent. GPTBot relates to training, while OAI-SearchBot decides whether you can appear in ChatGPT search answers. Block GPTBot and keep OAI-SearchBot allowed if you want to stay visible but opt out of training.

Does Google-Extended affect AI Overviews or rankings?

No. Google says Googlebot is the control for Search, including AI Overviews and AI Mode. Google-Extended covers AI training and grounding in some other Google systems, and Google says it does not affect Search inclusion or ranking.

Should I block ClaudeBot?

That is a content-policy choice, not an SEO one. Blocking ClaudeBot signals that your content should be left out of future training data. Claude-SearchBot and Claude-User handle search and live fetches, so keep those allowed if you want to stay visible in Claude's answers.

Does robots.txt stop ChatGPT-User and Perplexity-User?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person triggers the request, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says Claude-User honours it. If you must keep a fetcher out, use authentication instead of robots.txt.

Is llms.txt a substitute for robots.txt?

No. robots.txt controls access. llms.txt is an optional, unofficial summary of your best pages. Our llms.txt guide explains where it can help.

How long do robots.txt changes take to work?

OpenAI has said search-related changes can take around a day. Google says recrawling can take anywhere from several days to several months, depending on how often it decides a page needs refreshing.

Sources

• OpenAI: overview of crawlers

• Google: AI features and your website

• Google: crawlers and fetchers overview

• Anthropic: how its crawlers work and how to block them

• Perplexity: crawlers documentation

• Help Net Security: Cloudflare changes AI crawler access rules