What is ai.txt and How Do I Create One?

ai.txt is an emerging IETF standard that allows website owners to communicate machine-readable policies to AI crawlers and LLM agents.

How ai.txt differs from robots.txt

Feature	robots.txt	ai.txt
Purpose	Control crawling access	Control use of content (training, retrieval, citation)
Standard	Long-standing (1994)	Emerging IETF draft (2025)
Supported by	All major crawlers	GPTBot, ClaudeBot, Google-Extended (partial)
Granularity	Path-level	Use-case level (train, retrieve, cite)

Basic ai.txt template

Create a file at /ai.txt on your website root:

# ai.txt - AI Crawler Policies for example.com

# Allow all AI crawlers for retrieval (real-time answers)
User-agent: *
Retrieve: allow

# Block training on my content
Train: disallow

# Allow citation with attribution
Cite: allow

# Block specific paths from AI retrieval
Disallow: /private/

Directives explained

Directive	Meaning	Values
`Train`	Use content to train/fine-tune AI models	`allow`, `disallow`
`Retrieve`	Use content in real-time retrieval (RAG)	`allow`, `disallow`
`Cite`	Include citations/sources	`allow`, `disallow`, `require-attribution`
`Disallow`	Paths AI crawlers cannot access	URL paths

How to link ai.txt from HTML

Add to of your pages:

<link rel="ai-text" href="/ai.txt" type="text/plain">

Will AI crawlers respect ai.txt?

GPTBot has announced support for ai.txt (as of 2026)
Google-Extended has partial support (Training directives)
ClaudeBot supports the Train directive
PerplexityBot supports Retrieve and Cite

Not all crawlers support all directives yet, but the standard is rapidly being adopted.

Check your AI SEO Score — free, instant, no sign-up required

Get My Free AI SEO Score → View Full Audit Report