After a round of AI-search upgrades, owners usually ask one thing: did any model actually fetch the pages?
Analytics suites and search consoles do not list GPTBot or ClaudeBot on their own, yet these programs read the site every day. To turn “how much AI notices this site” into a number, you still need the most basic source: the server log. For a custom-built site, that log is a vital health signal during ongoing care.
Common AI crawlers
| Crawler | Organization | User-Agent token |
|---|---|---|
| GPTBot | OpenAI | GPTBot |
| ChatGPT-User | OpenAI (live answers) | ChatGPT-User |
| ClaudeBot | Anthropic | ClaudeBot or anthropic-ai |
| PerplexityBot | Perplexity | PerplexityBot |
| Google-Extended | Google (training) | Google-Extended |
| CCBot | Common Crawl | CCBot |
| Bytespider | ByteDance (Doubao) | Bytespider |
| Amazonbot | Amazon (Alexa) | Amazonbot |
A public roster of AI crawlers is kept at darkvisitors.com; check it about once a quarter.
Analyze logs with awk
Start with a few practical commands:
# Pull lines that mention major AI crawlersawk '/GPTBot|ClaudeBot|PerplexityBot|Google-Extended|CCBot/' access.log# Count hits for three crawlersawk '/GPTBot|ClaudeBot|PerplexityBot/ { if (/GPTBot/) gpt++ else if (/ClaudeBot/) claude++ else if (/PerplexityBot/) pplx++}END { print "GPTBot:", gpt, "Claude:", claude, "Perplexity:", pplx }' access.log# List paths hit most often by AI crawlersawk '/GPTBot|ClaudeBot/ { print $7 }' access.log | sort | uniq -c | sort -nr | head -20Visualize logs in GoAccess
GoAccess is a free, open-source log viewer. A basic report looks like this:
goaccess access.log --log-format=COMBINED -o ai-report.html --addr-spoof-key=GPTBot --addr-spoof-key=ClaudeBotThe HTML report shows daily trends, top paths, and how source addresses are spread.
Three core metrics
1. Monthly visit trend
On a healthy site, AI crawler visits tend to rise month by month.
| Month | GPTBot | ClaudeBot | Perplexity |
|---|---|---|---|
| Month 1 | Low | Low | Low |
| Month 2 | Medium | Medium | Medium |
| Month 3 | High | High | Medium-high |
A sudden drop deserves a look. Typical causes:
- The model already collected enough pages for now
- robots.txt is blocking crawlers by mistake
- 5xx errors are climbing
2. Crawl depth
How many levels do the bots actually reach? A healthy spread looks like this:
- Level 1 (home): nearly all visits
- Level 2 (categories and lists): about 60–80%
- Level 3 (inner pages and articles): about 30–50%
- Level 4 and deeper: under 10%
If level 3 is thin, internal links are probably too weak for bots to find deeper pages.
3. Error response share
# Count 404s seen by AI crawlersawk '/GPTBot|ClaudeBot/ && $9 == 404' access.log | wc -l# Count 5xx responses seen by AI crawlersawk '/GPTBot|ClaudeBot/ && $9 ~ /^5/' access.log | wc -lA high 404 count usually means dead links remain; a high 5xx count means the host is unstable. Repeated 5xx responses cause later crawl frequency to fall, so fix them at once.
Long-term ELK dashboards
Mid-size and large sites can generate gigabytes of logs each day. ELK fits that volume:
- Logstash parses the logs
- Elasticsearch indexes and stores them
- Kibana draws the dashboards
You can then track long-term trends, raise live alerts, and compare several sites.
Sample monitoring report
【AI crawler monthly note】2026-04GPTBot: 2,100 hits (up 33% vs prior month)ClaudeBot: 1,450 hits (up 32% vs prior month)Perplexity: 980 hits (up 36% vs prior month)Paths fetched most often:1. /insights/ai-ready-website-building-guide (387)2. /lp/ai-ready (256)3. /insights/schema-markup-web-design-guide (198)Follow-up:- /old-pricing/2024 returned 404, 47 times; add a 301Three common mistakes
- Watching only the raw hit count: follow the trend. A new site with few crawls is normal; a sudden drop on an older site is the warning.
- Blocking bots just because volume looks high: unless the pages are paid or confidential, blocking them hands AI visibility to competitors.
- Ignoring piled-up 4xx and 5xx responses: too many errors make crawlers come back less often, which is like closing the door yourself.
You cannot manage the unseen
Server logs are old-fashioned and still dependable. As AI search grows, their value is being noticed again. Spend about half an hour each month on an AI-crawler summary; it often shows site health earlier than search consoles or traffic tools. Pair that with the llms.txt setup guide so crawlers are steered on purpose, and the effect is easier to see.
A standing crawler dashboard is the clearest proof that these models keep paying attention to your content. Teams can use the monthly note to decide whether the content plan should change. See the solutions overview for related approaches.
