User-agent strings, verification methods, and a reference table for reading Nginx logs when you want to know which AI is reading you
TL;DR
Ten AI crawlers touch most sites in 2026: GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, and Amazonbot.
They split into three classes by purpose: training crawlers (building model corpora), retrieval crawlers (indexing for AI search), and on-demand fetchers (retrieving a specific URL in response to a live user query). Each class needs a different robots.txt policy and produces a different log signature.
Server logs are the only ground truth for AI crawler activity: platform dashboards and third-party tools lag by days or miss bots entirely. These ten user-agent strings turn raw Nginx access logs into a real-time window on which AI systems are reading your site and at what frequency.
Why Logs Beat Assumptions
I spent most of 2025 assuming AI crawler traffic was negligible on small sites. The reasoning seemed sound: a site with 50,000 monthly visitors shouldn’t see meaningful AI crawler activity, because why would an AI system bother indexing it? Then I actually read the Nginx access logs.
The GEO Stack framework treats crawlability as Layer 1, and log analysis is the only verification method that tells you whether Layer 1 is actually functioning for AI traffic, not just for Googlebot.
What I found: AI crawlers were the second-largest source of bot traffic after Googlebot, and a substantial portion of that was on-demand fetches: real-time retrievals triggered by actual user queries in ChatGPT, Perplexity, and Claude. Not training scraping. Not opportunistic indexing. Users asking AI a question, the AI fetching my page in response, within seconds of the question being typed.
The assumption was wrong because I was thinking about crawlers the way I’d learned to think about Googlebot: scheduled, predictable, background. Most AI crawler traffic doesn’t work that way. A meaningful chunk is synchronous with human questions. That changes what the logs tell you, and what you should do with them.
The ten crawlers below are the ones that produce enough log traffic to matter. Each entry has the exact user-agent string to grep for, the operator behind it, what it does, and how to verify it’s real.
Three Classes of AI Crawler
Before the list, the distinction that shapes how you read the logs.
Class differences go beyond purpose: AI crawlers diverge from Googlebot in crawl frequency, JavaScript handling, and robots.txt interpretation: the ten behavioural differences document what the Googlebot mental model gets wrong for each crawler class.
Training
Crawls content to build or refresh a model’s training corpus. Scheduled, broad, and indifferent to any particular user query. Examples: GPTBot, Google-Extended, ClaudeBot.
Retrieval
Crawls content to populate the retrieval index used at query time. Similar cadence to Googlebot: scheduled revisits, with priority weighted by update frequency. Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot.
On-demand
Fetches a specific URL in real time when an AI user asks a question that requires it. Synchronous with human input. Short bursts, often a single request per URL per query. Examples: ChatGPT-User, Perplexity-User, Claude-User.
The class a crawler belongs to determines three things: the robots.txt directive that controls it, the log pattern you’ll see, and whether blocking it has business consequences. Blocking a training crawler stops your content entering training data: which some sites want. Blocking an on-demand fetcher stops live AI users from ever seeing your page cited, which very few sites actually want.
The Ten AI Crawlers
Ordered by operator, grouped for quick reference. Each card has the exact user-agent string, the operator, purpose class, verification method, and a one-line Nginx log grep you can run right now.
01
GPTBot
OpenAI · Training
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
Purpose
Retrieves publicly accessible content for OpenAI’s foundation model training corpus. Does not contribute to ChatGPT’s real-time search responses.
Verification
OpenAI publishes an IP range JSON at openai.com/gptbot.json. Match the requesting IP against that file, or use reverse DNS: legitimate GPTBot traffic resolves to a subdomain under openai.com.
Log grep
grep 'GPTBot' /var/log/nginx/access.log | wc -lrobots.txt
User-agent: GPTBot followed by Disallow: / blocks all GPTBot crawling. Honoured by OpenAI as documented policy.
02
ChatGPT-User
OpenAI · On-demand
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)
Purpose
Fetches a specific URL when a ChatGPT user clicks a link, asks about a specific page, or uses browsing to answer a question. Synchronous with user input.
Verification
Same IP range file as GPTBot. The distinguishing signal in logs is the user-agent string and the request pattern: single-URL fetches rather than site-wide crawls.
Log grep
grep 'ChatGPT-User' /var/log/nginx/access.log
| awk '{print $7}' | sort | uniq -c | sort -rnOutput shows which specific URLs ChatGPT users are asking about in real time.
03
OAI-SearchBot
OpenAI · Retrieval
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
Purpose
Crawls and indexes content for ChatGPT Search: the retrieval layer behind ChatGPT’s answer generation when web grounding is enabled. Distinct from training and from on-demand browsing.
Verification
Same openai.com/searchbot.json IP range file. Note that this is a separate endpoint from GPTBot’s: they publish distinct ranges.
Why it matters
OAI-SearchBot access is a prerequisite for appearing in ChatGPT Search results. Blocking it blocks citation in the search surface: which is distinct from blocking training data via GPTBot.
04
PerplexityBot
Perplexity · Retrieval
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Purpose
Background retrieval crawler. Indexes content for Perplexity’s answer-generation retrieval layer. Scheduled revisits.
Verification
Perplexity publishes IP ranges at their documented endpoint. Reverse DNS on a legitimate PerplexityBot request resolves to a perplexity.ai subdomain.
Log grep
grep 'PerplexityBot' /var/log/nginx/access.log
| awk '{print $1}' | sort -u | headLists unique IPs seen: useful for spot-checking against the published IP ranges.
05
Perplexity-User
Perplexity · On-demand
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Purpose
On-demand fetcher triggered when a Perplexity user asks a question that requires retrieving a specific URL in real time. The most common AI crawler source of “user just asked about my page” traffic in 2026.
Log signature
Single-URL requests, often correlated in time with spikes in referral traffic from perplexity.ai. If you see Perplexity-User hit a specific URL and traffic from perplexity.ai rise within the same hour, the page has just been cited in a live answer.
06
ClaudeBot
Anthropic · Training
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Purpose
Retrieves content for Claude model training. Analogous to GPTBot’s role at OpenAI: training corpus construction, not live retrieval.
Verification
Anthropic publishes an IP range file; reverse DNS resolves to anthropic.com subdomains. The contact email in the user-agent, claudebot@anthropic.com, is a unique identifier absent from spoof user-agents.
robots.txt
User-agent: ClaudeBot followed by Disallow: / blocks training crawl. Does not affect Claude-User or Claude-SearchBot.
07
Claude-User
Anthropic · On-demand
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)
Purpose
On-demand fetcher triggered when a Claude user asks about a specific URL, or when Claude uses browsing in a conversation. Real-time, synchronous with user input.
Log signature
Single-URL fetches with no follow-up crawl. If you see Claude-User hit a page with no preceding or following requests from the same operator, a user is almost certainly asking Claude about that specific page right now.
08
Claude-SearchBot
Anthropic · Retrieval
User-agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)
Purpose
Retrieval crawler for Claude’s web search and research tools. Scheduled crawls, populates the retrieval index consulted when Claude uses web search during a conversation.
Why it matters
The three Anthropic crawlers correspond to three distinct functions. Blocking ClaudeBot preserves the option to opt out of training; keeping Claude-SearchBot and Claude-User allowed keeps the site visible to Claude’s live users. Conflating them blocks too much or too little.
09
Google-Extended
Google · Training
User-agent
No distinct user-agent. Google-Extended is a robots.txt product token, not a separate crawler: Googlebot handles the actual fetching, and the token tells Google whether to use the content for Gemini and Vertex AI training.
Purpose
Controls whether content crawled by Googlebot is also used to train Gemini and Vertex AI generative models. The crawler itself is still Googlebot; Google-Extended is the opt-out mechanism.
robots.txt
User-agent: Google-Extended
Disallow: /Blocks training use without affecting Google Search indexation. This is the only way to opt out of Gemini training while remaining in Google Search.
Log grep
Not applicable, no distinct user-agent appears in logs. Opt-out works through robots.txt compliance, not IP-level blocking.
10
Amazonbot
Amazon · Training & retrieval
User-agent
Mozilla/5.0 (Linux; ... ) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Purpose
Amazon’s general-purpose crawler, used for Alexa content sourcing and increasingly for Amazon’s generative AI products. Crawls are scheduled rather than on-demand.
Verification
Reverse DNS resolves to crawl.amazonbot.amazon subdomains. Amazon publishes IP ranges in their developer documentation.
Log grep
grep -iE 'Amazonbot|AlexaBot' /var/log/nginx/access.log | wc -lThe Ten at a Glance
One reference table. Bookmark this if you want a quick lookup for what each user-agent string means and whether blocking it has consequences.
| # | Crawler | Operator | Class | If you block it |
|---|---|---|---|---|
| 01 | GPTBot |
OpenAI | Training | Content excluded from GPT training corpora |
| 02 | ChatGPT-User |
OpenAI | On-demand | ChatGPT users asking about your page see no content |
| 03 | OAI-SearchBot |
OpenAI | Retrieval | Not cited in ChatGPT Search |
| 04 | PerplexityBot |
Perplexity | Retrieval | Not indexed for Perplexity answers |
| 05 | Perplexity-User |
Perplexity | On-demand | Perplexity users querying your URL see nothing |
| 06 | ClaudeBot |
Anthropic | Training | Content excluded from Claude training corpora |
| 07 | Claude-User |
Anthropic | On-demand | Claude users can’t retrieve your pages mid-conversation |
| 08 | Claude-SearchBot |
Anthropic | Retrieval | Not cited in Claude web search responses |
| 09 | Google-Extended |
Training | Opt-out of Gemini and Vertex AI training | |
| 10 | Amazonbot |
Amazon | Training | Excluded from Amazon AI products and Alexa |
Four Log Queries That Actually Tell You Something
Four Nginx one-liners that turn raw access logs into usable AI crawler intelligence. Each assumes the default Nginx combined log format at /var/log/nginx/access.log: adjust the path if yours differs.
After confirming AI crawler visits via logs, the next diagnostic is a GEO audit: the ten GEO audit checks identify which structural failures are most likely to suppress citation despite confirmed crawling.
1. Count all AI crawler requests in the last 24 hours.
grep -iE 'GPTBot|ChatGPT-User|OAI-SearchBot|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Claude-SearchBot|Amazonbot'
/var/log/nginx/access.log
| awk -v d="$(date -d '24 hours ago' '+%d/%b/%Y:%H')" '$4 > "["d'
| wc -l2. Which pages are on-demand fetchers hitting right now? This one tells you which URLs are being asked about in live AI conversations.
grep -iE 'ChatGPT-User|Perplexity-User|Claude-User'
/var/log/nginx/access.log
| awk '{print $7}' | sort | uniq -c | sort -rn | head -203. Which AI crawlers visit most often? Distribution across operators over the whole log file.
grep -iE 'GPTBot|ChatGPT-User|OAI-SearchBot|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Claude-SearchBot|Amazonbot'
/var/log/nginx/access.log
| grep -oiE 'GPTBot|ChatGPT-User|OAI-SearchBot|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Claude-SearchBot|Amazonbot'
| sort | uniq -c | sort -rn4. What is the 4xx/5xx rate for AI crawlers? A high error rate for a specific crawler means it’s hitting blocked pages, broken redirects, or pages behind auth. Every error is content that will never reach the retrieval index.
grep -iE 'GPTBot|PerplexityBot|ClaudeBot|Claude-SearchBot|OAI-SearchBot'
/var/log/nginx/access.log
| awk '{print $9}' | sort | uniq -c | sort -rnOn user-agent spoofing. User-agent strings are trivial to fake. A meaningful fraction of traffic claiming to be GPTBot or PerplexityBot is not, it’s scrapers and competitive intelligence tools using the AI crawler user-agent to avoid being blocked. For any count above low hundreds per day, verify a sample of IPs against the operator’s published range file before acting on the numbers.
What to Put in robots.txt
Three common policies. Pick the one that matches your posture, not whichever one sounds most cautious by default.
Crawl policy changes affect citation indirectly and with a lag: a citation measurement run before and after any robots.txt change is the only reliable way to quantify the effect.
Policy A — allow everything. Default. Suitable for content that benefits from being in both training data and live AI answers. This is the right policy for most publishers who want to be cited.
# robots.txt: allow all AI crawlers (default)
# No AI-specific directives needed.Policy B — opt out of training, allow retrieval and on-demand. Keeps the site visible to live AI users and in AI search results, but excludes content from model training corpora. The compromise most publishers actually want.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /Policy C — block all AI. Nuclear option. Removes the site from every AI retrieval and training pipeline. Suitable for content with licensing restrictions or sensitivity that makes AI exposure unacceptable. The side-effect is total invisibility in AI-generated answers.
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /The GEO Lab runs Policy A. The whole point is to be cited.
Quick reference for the trade-offs:
| Policy | Training | Retrieval | On-demand | Best for |
|---|---|---|---|---|
| A — Allow all | ✓ | ✓ | ✓ | Publishers who want maximum citation potential |
| B — No training | ✗ | ✓ | ✓ | Sites that want live AI visibility without training exposure |
| C — Block all | ✗ | ✗ | ✗ | Content with licensing restrictions or legal sensitivity |
Frequently Asked Questions
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s training crawler: it retrieves content for model training and pre-training corpus construction. ChatGPT-User is the on-demand fetcher that retrieves a specific URL when a ChatGPT user asks about it in a conversation. The two have different user-agent strings, different crawl patterns, and different robots.txt semantics. Blocking one does not block the other, and treating them as a single entity in log analysis leads to misdiagnosed crawl behaviour.
How do I verify an AI crawler is real and not a spoof?
Three methods, in order of reliability: reverse DNS lookup on the requesting IP should resolve to a subdomain of the operator (for example openai.com or anthropic.com), then forward DNS on that hostname should resolve back to the same IP; match the IP against the operator’s published IP range JSON file; and check request headers for operator-specific signatures. Reverse-and-forward DNS verification is the standard Google uses and is the most reliable for operators that publish verification endpoints.
What user-agent does Perplexity use?
Perplexity uses two distinct user-agents. PerplexityBot is the background crawler that indexes content for Perplexity’s retrieval system. Perplexity-User is the on-demand fetcher triggered when a user asks Perplexity a question that requires retrieving a specific URL in real time. Both contain the literal string “Perplexity” in the user-agent header, but their crawl behaviour differs substantially: PerplexityBot crawls at scheduled intervals, while Perplexity-User fetches within seconds of a user query.
Do AI crawlers execute JavaScript?
Most AI crawlers do not execute JavaScript at crawl time. GPTBot, PerplexityBot, and ClaudeBot retrieve the raw HTML source and parse it without running client-side scripts. A minority of on-demand fetchers, typically the per-query browsing agents, may render JavaScript, but their behaviour is inconsistent across operators and model versions. Content that relies on client-side rendering is effectively invisible to the majority of AI crawler traffic.
How do I block an AI crawler without blocking legitimate traffic?
The correct layer is robots.txt, using the crawler’s documented user-agent string and a Disallow directive. Firewall-level blocks risk collateral damage if the crawler shares IP ranges with other services. User-agent matching at the web server layer is effective for crawlers that honour robots.txt and legal for crawlers that do not, but it still requires the exact documented user-agent string: a partial match will miss variants.