AI Crawler robots.txt Analyzer
Paste your robots.txt to see which AI crawlers you're allowing or blocking — across OpenAI, Anthropic, Google, Perplexity, Apple, Meta, ByteDance, Amazon, and more. Most sites silently allow every AI crawler without realizing the trade-offs. This tool tells you exactly which bot does what so you can decide on purpose.
Paste your robots.txt below. We'll classify which AI crawlers you're allowing, blocking, or silently passing through — and explain the trade-off for each. Most site owners haven't audited this; blocking the wrong bot quietly removes your business from AI answers without you noticing.
Three canonical configurations. Copy any one and paste it as your full robots.txt at your site root.
Maximize AI discoverability. Recommended for most businesses competing for AI search visibility — most don't realize their content is what makes them findable in AI answers.
The 'have your cake and eat it too' option — your content can still be cited in ChatGPT / Claude / Perplexity answers, but won't be used to train future models. Recommended if you have IP concerns about training but want AI search visibility.
Maximum exclusion — your content won't appear in any AI answer, citation, or training corpus. Rarely the right call unless you have specific legal/IP requirements. Most businesses lose discoverability for no upside.
The most common misconception: people block GPTBot thinking it stops ChatGPT from "using" their content — but GPTBot is only for training. ChatGPT citations come from OAI-SearchBot and ChatGPT-User. Same trap with ClaudeBot (training) vs Claude-User/ Claude-SearchBot (citations). If you want AI visibility, the search/citation bots are the ones to allow — regardless of how you feel about training.
Get a free AI Search Readiness Audit
robots.txt is the access permissions layer. The audit also covers JS rendering, schema markup, llms.txt, SSR vs CSR, and how ChatGPT / Claude / Perplexity actually cite your business compared to competitors. 48-hour turnaround. No signup. No sales pitch.
Request your free audit →