UltraScout AI › Learn › How do I check if AI bots can crawl my website?

How to check if AI bots can crawl your website

AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) need explicit permission to access your content. Here's how to check:

Method 1: Manual inspection

1. Check robots.txt — Go to https://yourdomain.com/robots.txt. Look for lines like:

- User-agent: GPTBot followed by Disallow: / (blocked) or Allow: / (allowed)

- User-agent: ClaudeBot

- User-agent: Google-Extended

1. Check meta tags — View page source and search for:

- (blocks AI)

-

1. Check for ai.txt — Try https://yourdomain.com/ai.txt or /llms.txt (emerging standards).

Method 2: Automated tool (recommended)

UltraScout's free AI SEO Score does all this in 10 seconds:

• Parses your robots.txt for all major AI user-agents
• Detects AI-blocking meta directives
• Checks for ai.txt and llms.txt presence
• Provides a crawlability score from 0-100

What the results mean

No signup required. Instant results.