78 days of IP-verified log analysis on a single domain. The raw ratio is 1:9.5, the verified-AI ratio is 1:74.1, and 72% of bot-claimed llms.txt requests fail IP verification.
Do AI bots actually fetch llms.txt? On thegeolab.net, 7 verified AI-bot fetches of llms.txt occurred in 78 days, against 2,292 RSS feed requests. The raw llms.txt-to-RSS ratio is 1:9.5. After IP verification, 72% of bot-claimed llms.txt requests were spoofed. GPTBot is the only verified bot that fetched llms.txt more than once. The file’s primary audience is the SEO industry auditing whether the file exists.
Verdict first. Do AI bots actually use llms.txt? Hank Azarian claimed AI bots fetch RSS feeds roughly 100 times more often than llms.txt. That ratio is directionally correct but the magnitude is wrong by an order at the raw layer. On this site, across 78 days of IP-verified log analysis on the GEO Lab research log, the raw ratio is 1:9.5 and the verified-AI ratio is 1:74.1. More importantly, the gap between raw counts and verified counts reveals that most published llms.txt adoption metrics rest on user-agent strings that do not survive IP verification.
Three ratios, one story
Every layer of verification makes llms.txt look worse relative to RSS. That spread is the finding.
| Layer | llms.txt | RSS feed | Ratio |
|---|---|---|---|
| Raw (all requests, all clients) | 241 | 2,292 | 1:9.5 |
| UA-classified AI bots | 25 | 535 | 1:21.4 |
| IP-verified AI bots | 7 | 519 | 1:74.1 |
The raw ratio (1:9.5) reflects what the server logs see. The UA-classified ratio (1:21.4) isolates traffic claiming an AI-bot identity. The IP-verified count (7 confirmed AI fetches of llms.txt in 78 days) is what survives CIDR and rDNS verification. Feed-side IP verification was run on all AI-claimed feed requests: 519 of 535 verified, a 3.0% spoof rate. The verified ratio is 1:74.1, far steeper than the UA-classified 1:21.4, because almost all AI feed traffic is genuine while most AI llms.txt traffic is spoofed.
Most published studies stop at layer one or two. The 191-day/900-domain study found roughly 6 llms.txt requests per day, with zero from genuine AI bots after manual review. Buytaert reported roughly 5,000 llms.txt fetches across 400 million total requests. None of these applied IP verification to the llms.txt fetchers. The gap between our UA-classified count (25) and our IP-verified count (7) suggests their numbers would contract too.
Who actually fetches llms.txt
The per-bot table is the citable artifact. Verified fetchers are exclusively OpenAI and one Microsoft bot. ClaudeBot crawls this site heavily (5,975 IP-verified requests over the window) and has never once fetched llms.txt.
| Bot | llms.txt fetches | Method | Total site requests | llms.txt share |
|---|---|---|---|---|
| GPTBot | 4 | CIDR-verified | 4,277 | 0.094% |
| OAI-SearchBot | 2 | CIDR-verified | 4,099 | 0.049% |
| bingbot | 1 | rDNS-verified | 3,036 | 0.033% |
| Verified total | 7 |
GPTBot is the only verified AI bot that fetched llms.txt more than once. Four visits in 78 days: roughly fortnightly. That is the entire verified signal.
Going deeper? The GEO Field Manual covers bot verification methodology, citation measurement protocols, and the full audit framework used in this analysis.
PerplexityBot and the spoofing problem
Five requests carried PerplexityBot’s user-agent string. Perplexity publishes verification ranges at perplexitybot.json (8 prefixes, 18 addresses, creationTime February 2025). All five failed: the IPs trace to hosting providers (hostodo.com, shoplachica.com) and a bug-bounty scanner (bugbounty-dash), none inside Perplexity’s published ranges. They are spoofed, not unattributable.
The spoofing picture is broader. Eighteen llms.txt requests carried bot user-agent strings that IP verification disproved: six fake Googlebot, five fake PerplexityBot, four fake GoogleOther, two fake GPTBot, and one fake bingbot. All came from IPs outside the operators’ published CIDR ranges. That is 18 of 25 bot-claimed requests (72%) failing verification on a single file. On this site’s feed endpoints, the UA-only spoofing rate was 3.0%. llms.txt, as a newer and more niche file, attracts disproportionate scanner attention.
The file’s real audience
Of 241 llms.txt requests, 216 carried no bot user-agent or carried a non-AI-bot UA. These are SEO auditing tools, llms.txt scanning services, and researchers probing whether the file exists. The file’s primary audience is the industry auditing the file.
I expected the feed to be the one place actual humans still showed up. They did not. Of 2,292 feed requests (WordPress RSS endpoints, self-hits excluded), 11 carried RSS-reader user agents, and all 11 were FeedBot/1.0, itself an automated fetcher rather than a person with a reader app. The feed is bot infrastructure: AI crawlers (GPTBot 202, ClaudeBot 139, OAI-SearchBot 56), SEO bots (AhrefsBot 214, MJ12bot 97, DotBot 58), and unclassified crawlers with browser-like user agents. The AI-crawler counts sit inside a population measured at 3.0% spoofing; the SEO-bot counts are UA-classified only. Both files serve machines, not people.
Infrastructure texture
The nginx access_log directive that recorded Accept: text/markdown content-negotiation requests (markdown-requests.log) was removed on 3 April 2026 and restored on 20 July 2026, a gap of 3.5 months. Content negotiation and llms.txt are complementary discovery mechanisms; for this window the analysis measures path-based fetches only, and the restored log resumes negotiation capture from 20 July.
llms-full.txt has never existed on this domain. Zero requests for that path appeared in the 78-day window. Deploying it, then measuring pickup, is the planned follow-up to this note.
Methods
Window. 2 May to 18 July 2026, 78 complete days. Source: 79 nginx access log files (access.log through access.log.78.gz), covering every request to thegeolab.net in the window.
Canonical counting rule. llms.txt: exact path /llms.txt, all HTTP status codes, requests originating from the server itself (self-hits) were excluded. This yields 241 requests. Path variants excluded: /.well-known/llms.txt (12 requests, all 404, path does not exist), /llms.txt. trailing-dot probes (6), RankMath admin-ajax keyword checks containing “llms.txt” in a query parameter (3). Feed: WordPress RSS endpoint pattern (/feed/, /*/feed/, /comments/feed/, /feed/(atom|rss|json)/), self-hits excluded. Implementation: re.search(r'/feed/?$|/feed/', path), which matches //feed/ double-slash paths and /feed without trailing slash (both within the /*/feed/ glob description). This yields 2,292 requests.
Count reconciliation. An earlier in-session analysis of the same nominal window reported 372 llms.txt and 2,127 feed requests. The llms.txt figure of 372 is not reproducible from these logs; the broadest defensible count (all path variants, all statuses, including self-hits) is 274. The feed figure of 2,127 is consistent with a slightly narrower RSS-path filter than used here (2,292). The discrepancy in the llms.txt count is unresolved; possible causes include a different path-matching rule, a window-boundary difference, or a counting error in the earlier session. This note uses only the reproducible counts.
| Reconciliation | Earlier session | This analysis | Cause |
|---|---|---|---|
| llms.txt | 372 | 274 (broad) / 241 (canonical) | Not reproducible; possible path or window difference |
| RSS feed | 2,127 | 2,292 (canonical) | Consistent; minor path-filter difference |
Bot verification. Every llms.txt-fetching IP was verified through bot_verify.py (commit 86e18dc), the same tool used in the GEO Brand Citation Index pipeline. CIDR verification for OpenAI (3 range files) and Google (4 range files, re-resolved to /static/crawling/ipranges/ after the June 2026 URL migration). Forward-confirmed rDNS for Microsoft/bingbot. Perplexity verified via CIDR against perplexitybot.json (creationTime 2025-02-07, 8 prefixes, 18 addresses). Anthropic/ClaudeBot verified via CIDR against claude.com/crawling/bots.json (creationTime 2026-05-01, 20 prefixes, 1,043 addresses). Feed-side IP verification was run on all AI-claimed feed requests (519 verified, 16 spoofed, 3.0% spoof rate). Note: the cited commit 86e18dc of bot_verify.py did not consult the Anthropic or Perplexity lists; the corrected figures use the vendor-published ranges directly.
Single-domain caveat. All figures describe one domain (thegeolab.net) over 78 days. This is a technical site in the GEO/SEO niche with an established llms.txt deployment. Generalisation to other domains, verticals, or time windows requires separate measurement, as the noise-floor analysis demonstrated for citation-rate studies.
Status-code split (/llms.txt). 200: 206 requests. 301: 27 (http-to-https or trailing-slash redirect, each followed by a 200). 404: 17 (pre-deployment probes or transient errors). 206: 1 (range request). The file itself serves 200.
Comparison with published studies
| Metric | thegeolab.net | Published study | Classification method |
|---|---|---|---|
| llms.txt as % of total requests | 0.015% | Buytaert: ~0.001% | Both UA-agnostic (all clients) |
| llms.txt requests per day | 3.1 total (0.09 verified-AI) | 191-day study: ~6/day, 0 genuine AI | Ours IP-verified; study used manual review |
Key GEO Lab Takeaway
llms.txt adoption by AI bots is near-zero when measured with IP verification: 7 verified fetches in 78 days, and GPTBot is the only bot that returned more than once.
72% of bot-claimed llms.txt requests are spoofed. Any study counting llms.txt adoption without IP verification overstates real demand.
RSS feeds receive 9.5x more total traffic and 74.1x more verified-AI traffic than llms.txt. For AI discoverability, the feed your CMS already generates outperforms the file the industry is debating.
Want to see how AI engines actually cite your content?
The GEO Lab measures what AI search retrieves, cites, and ignores. Read the methodology behind the numbers.
What is GEO?Corrections (23 July 2026, updated 23 July 2026):
The bot-verification pipeline (bot_verify.py, commit 86e18dc) did not consult two published vendor IP lists: Anthropic’s (bots.json) and Perplexity’s (perplexitybot.json, which has existed since February 2025). This inflated the unverifiable bucket and understated the spoof rate. With both lists applied, all 5 PerplexityBot-claiming llms.txt requests are confirmed spoofed (IPs trace to hosting providers and a bug-bounty scanner). The corrected llms.txt spoof rate is 72% (18 of 25), not the originally published 52% (13 of 25). The verified AI fetch count (7) is unchanged.
The original version also stated Google-Extended “crawls the site daily.” Google-Extended is a robots.txt opt-out token, not a crawler; it makes zero HTTP requests. That claim was wrong in premise and has been removed. ClaudeBot verification, previously broken, now uses Anthropic’s published list; 93.9% of ClaudeBot-claiming requests verified as genuine.
The feed spoofing rate was corrected from the original “under 1%” to 3.0% (16 of 535). Feed-side IP verification, originally not run, was completed: 519 of 535 AI-claimed feed requests verified, giving a verified llms.txt-to-feed ratio of 1:74.1. The following figures reproduced exactly on recomputation and are unchanged: 241 llms.txt requests, 2,292 feed requests, 1:9.5 raw ratio, 7 verified AI fetches. The AI-classified ratio shifted from 1:21.6 to 1:21.4 when the feed count was corrected from 541 to 535.