When pages are used to train models, or condensed by chat assistants, readers often never return to the original site. That fear of wasted effort leads many owners to consider blocking AI crawlers outright.
Since about 2024, this has been one of the most common strategy questions from site owners. The usual trap is copying a peer who blocks everything, which can quietly hurt your own visibility.
Put simply, the choice is not a blanket block or a blanket allow, but a decision based on the business model. The notes below are a practical test you can apply by industry.
Major AI crawlers and uses
| Crawler | Organization | Main use |
|---|---|---|
| GPTBot | OpenAI | Collects training material for ChatGPT |
| ChatGPT-User | OpenAI | Fetches live results for ChatGPT |
| ClaudeBot | Anthropic | Collects training material for Claude |
| anthropic-ai | Anthropic | Fetches live results for Claude |
| PerplexityBot | Perplexity | Searches live pages and cites sources |
| Google-Extended | Trains Gemini, separate from Googlebot | |
| CCBot | Common Crawl | Builds an open corpus used by many models |
| Bytespider | ByteDance | Collects training material for Doubao |
Pros and cons of blocking
Benefits of staying open
- Exposure from citations: mentions in ChatGPT, Claude, or Perplexity put the brand in front of more people
- An early search advantage: once AI search is mainstream, sites models already know tend to lead
- A mark of authority: being a cited source is itself a form of industry recognition
- Traffic that arrives later: branded queries often rise about three to twelve months after citations begin
Benefits of blocking bots
- Protect paid material: stop subscription articles and member areas from being summarized for free
- Keep unique data: research, industry figures, and exclusive analysis stay in-house
- Cut crawl load: high-traffic sites can save bandwidth and server capacity
- Reduce compliance risk: healthcare and finance sometimes need tighter control
Strategies for three industries
Paid content brands
Approach: block only paid areas
Let crawlers read the homepage, titles, and summaries, but refuse full articles, white papers, and databases:
# Limit paid pathsUser-agent: GPTBotUser-agent: ClaudeBotUser-agent: PerplexityBotUser-agent: Google-ExtendedDisallow: /premium/Disallow: /members/Disallow: /reports/User-agent: *Allow: /B2B industry and services
Approach: allow by default
For business buyers, being named by a model is often one of the most valuable ways to win leads. Many industrial purchasers used to compare suppliers in search for days; a summary that names you is an unpaid acquisition channel. Blocking it hands those inquiries to rivals who remain citable.
Unless healthcare, finance, or a similar rule forbids it, keep the corporate site open:
User-agent: *Allow: /Disallow: /admin/Disallow: /api/Online retail
Approach: allow by default
Product details need to be found, and a model citation is free promotion. Blocking these crawlers brings online stores almost no benefit:
User-agent: *Allow: /Disallow: /admin/Disallow: /cart/Disallow: /checkout/Disallow: /orders/Full robots.txt samples
Sample 1: allow most paths
User-agent: *Allow: /Disallow: /admin/Disallow: /api/Disallow: /private/Sitemap: https://yoursite.com/sitemap.xmlSample 2: block paid areas
# Training crawlers: skip paid pathsUser-agent: GPTBotDisallow: /premium/Disallow: /members/User-agent: ClaudeBotDisallow: /premium/Disallow: /members/User-agent: anthropic-aiDisallow: /premium/Disallow: /members/User-agent: PerplexityBotDisallow: /premium/User-agent: Google-ExtendedDisallow: /premium/# Other crawlersUser-agent: *Allow: /Disallow: /admin/Disallow: /api/Sitemap: https://yoursite.com/sitemap.xmlSample 3: block nearly all
User-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: anthropic-aiDisallow: /User-agent: PerplexityBotDisallow: /User-agent: Google-ExtendedDisallow: /User-agent: CCBotDisallow: /# Still allow Googlebot so classic search is unaffectedUser-agent: GooglebotAllow: /User-agent: *Disallow: /admin/Caution: a total block removes the site from AI search, and it fits only rare cases such as paid-content sites, official secrets, or medical privacy.
How to verify settings
- Fetch the file directly:
curl https://yoursite.com/robots.txt - Use Google's Robots.txt Tester (inside Search Console)
- Check server records: after one or two weeks, see whether GPTBot visits drop if you blocked it; for a method, see how to read AI crawlers in server logs
Three common owner mistakes
- Blocking Googlebot as well: that can wipe classic rankings, because GPTBot is not Googlebot. Some owners rejected every bot and watched rankings collapse within about a week.
- Incomplete paths:
Disallow: /premium(no trailing slash) also blocks paths such as/premium-x. Always confirm the file in Search Console's tester. - Relying on robots.txt while pages stay public: polite crawlers stop; others may still fetch. Paid content needs a server-side permission check. This file alone does not lock the door.
Start with full access
Unless there is a clear business reason to refuse access, starting fully open is the lower-risk choice. Citation gains belong to early movers; blocking now means giving that up. If the site is already open and still uncited, review seven common reasons models skip a site and check each one.
Switching later to partial blocks usually takes only minutes in robots.txt; missing the early window can cost years of visibility.
Business choice over tech
How robots.txt is written is not mainly an engineering detail. It is a question for the owner: how do you want the site to be discovered?
Before deciding, ask three things: (1) Is the core content public marketing, or visible only after payment? (2) Are key rivals refusing these crawlers? (3) Can you accept nearly vanishing from AI search for several years? Once those are answered, the robots.txt direction becomes clear.
