SV Scale Visibility.
Free Tool · Scale Visibility

AI Crawler robots.txt Analyzer

Paste your robots.txt to see which AI crawlers you're allowing or blocking, across OpenAI, Anthropic, Google, Perplexity, Apple, Meta, ByteDance, Amazon, and more. Most sites silently allow every AI crawler without realizing the trade-offs. This tool tells you exactly which bot does what so you can decide on purpose.

Paste your robots.txt below. We'll classify which AI crawlers you're allowing, blocking, or silently passing through, and explain the trade-off for each. Most site owners haven't audited this; blocking the wrong bot quietly removes your business from AI answers without you noticing.

·
Generate your corrected robots.txt

Pick the AI-crawler policy you want. Paste your current robots.txt above and we'll merge the policy into it; with nothing pasted you'll get a clean starter file.

Starter robots.txt
# robots.txt: AI-crawler policy applied with Scale Visibility's free tool
# https://www.scalevisibility.com/tools/robots-txt-analyzer/
# Existing non-AI rules are preserved as-is; the AI-crawler rules below
# reflect your chosen policy. Place this file at https://yourdomain.com/robots.txt

# ── AI crawlers: allow all (training + search/citation) ──
# Explicit Allow guarantees these bots are permitted even if your
# User-agent: * rule below is restrictive.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: YouBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: FacebookBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: omgili
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Diffbot
Allow: /

# Default for all other (non-AI) crawlers
User-agent: *
Allow: /
Why this matters for AI search

The most common misconception: people block GPTBot thinking it stops ChatGPT from "using" their content, but GPTBot is only for training. ChatGPT citations come from OAI-SearchBot and ChatGPT-User. Same trap with ClaudeBot (training) vs Claude-User/ Claude-SearchBot (citations). If you want AI visibility, the search/citation bots are the ones to allow, regardless of how you feel about training.
Beyond robots.txt: is your whole site AI-ready?

Get the full AI Search Readiness Audit

robots.txt is the access permissions layer — it decides whether the assistants can read you at all. The audit also covers JS rendering, schema markup, llms.txt, SSR vs CSR, structured data coverage and page speed.

The free score needs no account: your technical readiness number in seconds, plus, if you want it, one real buyer-intent query put live to ChatGPT, Claude, Gemini and Perplexity, so you can read their actual answers and see whether you get named. The full audit runs five queries across all four surfaces, shows which competitors get named in your place and which sites the assistants built those answers from, checks what they get wrong about your confirmed business facts (hours, phone, address), and hands you every fix paste-ready. $49, one-time founding price.

Start with your free score →

FAQ

Frequently asked questions

Add a directive for each crawler's user-agent, for example User-agent: GPTBot followed by Disallow: /. Paste your current robots.txt into the analyzer above and it will show which AI crawlers you allow or block today, then generate a corrected file for allow-all, search-only, or full-block while keeping your existing sitemaps and rules intact.

The ones that matter most: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI training), Applebot-Extended (Apple), Bytespider (ByteDance), Meta-ExternalAgent (Meta), and Amazonbot. The analyzer checks for all of these and more.

Not exactly, and this is the most common mistake. GPTBot is used mainly to gather training data. OAI-SearchBot is the crawler behind ChatGPT's live search and browsing. If you want to stay visible in ChatGPT search while opting out of training, you'd allow OAI-SearchBot but disallow GPTBot. Blocking the wrong one can quietly remove you from AI answers. The two-crawler trap is covered in more depth in our guide to llms.txt and robots.txt.

It's a trade-off. Blocking AI crawlers protects your content from training use, but it also makes you invisible when buyers ask AI assistants for recommendations in your field. For most businesses that want to be found, allowing the search-oriented crawlers (and deciding on training crawlers separately) is the right call. Run the AI Visibility Check to see whether AI assistants currently name you.

At the root of your domain: yourdomain.com/robots.txt. If you don't have one, every crawler is allowed by default. Once you've set your policy, the AI Search Readiness Audit checks robots.txt alongside the rest of your AI-visibility signals.

More free tools

Other AI-visibility tools

AI Visibility Check

See whether ChatGPT, Claude, Gemini, and Google AI Overviews name your business.

Schema Markup Generator

Generate paste-ready JSON-LD for LocalBusiness, Organization, FAQ, and more.

llms.txt Generator

Build a properly-formatted llms.txt file to guide LLMs through your site.

Entity & Knowledge Graph Check

Score how clearly your business exists as an entity AI assistants can trust and name.

PageSpeed + Revenue Impact Calculator

Test Core Web Vitals and see what slow load times cost you in conversions.
← Browse all free tools