What the AI Crawler Checker Tests
The checker reads the public robots.txt file for the submitted website and evaluates crawler directives for common AI and search user-agents. It helps surface an accidental block that could prevent a crawler from requesting content.
AI Crawlers and Search Crawlers
Different services use different user-agents and may use them for different purposes. A rule that allows Googlebot does not automatically describe access for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended, so each relevant directive should be reviewed separately.
What an Allowed Result Means
Allowed means the inspected robots.txt rules do not block that crawler for the tested location. It does not mean the service will definitely crawl, index, train on, quote or cite the page. Those decisions are controlled by the external platform.
When to Review robots.txt
Check crawler access after a redesign, migration, staging-to-production launch, security change or robots.txt edit. An unintended Disallow rule can remain unnoticed because the website still looks normal to human visitors.
Free to Use Without an Account
This SEO Mentor tool is available without registration, subscriptions or credit packs. It uses the information you enter or public website data needed for the check and does not turn unavailable third-party metrics into invented numbers.