UltraScout AI › Learn › How to test robots.txt for AI crawlers?

How to test robots.txt for AI crawlers

Your robots.txt file controls which crawlers can access your site. Here's how to test it for AI bots specifically.

Method 1: Manual inspection

1. Visit https://yourdomain.com/robots.txt
2. Search for these user-agents:

User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: anthropic-ai

1. For each, check what follows:

• Disallow: / — Blocked from all content
• Allow: / — Allowed full access
• Disallow: /private/ — Blocked only from specific paths
• (Missing) — Follows default (usually allowed)

Method 2: UltraScout automated test (fastest)

UltraScout's free AI SEO Score automatically:

• Fetches your robots.txt
• Parses all User-agent directives
• Identifies rules for all major AI crawlers
• Shows you at a glance which are allowed vs blocked

Common issues to watch for

Example: Good robots.txt for AI

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /private/