🤖 Free Tool · No account needed
Robots.txt AI Crawl Checker
Enter any domain and instantly see which AI crawlers are allowed or blocked — with a ready-to-copy fix for every issue found.
ROBOTS.TXT AI CRAWL CHECKER
GEOLAB.NET
No login required. Works on any public domain.
Fetching robots.txt…
How to use this tool
1
Enter your domain
Just the domain — no path needed. The tool fetches your robots.txt from the standard location at yoursite.com/robots.txt.
2
Review AI crawler status
Each major AI crawler is checked: GPTBot (OpenAI), PerplexityBot, ClaudeBot (Anthropic), Google-Extended, Bingbot, and Applebot. Status is Allowed, Blocked, or Default (no explicit rule).
3
Understand “Default” status
A crawler marked Default has no specific rules — it follows your wildcard User-agent: * rules. If you have Disallow: / as a wildcard, all Default crawlers are blocked. Check your wildcard rules carefully.
4
Copy the fixes and update robots.txt
Each blocked crawler gets a ready-to-copy Allow rule. Add these to your robots.txt file and verify at yoursite.com/robots.txt within a few minutes of deploying.
5
Re-run after making changes
Once you’ve updated robots.txt, re-run this tool to confirm the changes are live. AI crawlers typically pick up robots.txt changes within 24–48 hours.
Technical GEO Series
Fix your crawler access layer
6 emails on Technical GEO — robots.txt, crawl budget, Core Web Vitals, and the server-level configuration that directly affects AI citation probability.
Have questions about this topic? Contact The GEO Lab · Return to homepage