Free AEO Utility

Generate a robots.txt that keeps AI crawlers open.

Configure access for ChatGPT Search, Perplexity, Google AI Overviews, Claude, Grok, Bing, and standard search bots, then export a deployment-ready file instantly.

No sign-in
Adjust crawler permissions and copy the result immediately.
AI crawler coverage
OpenAI, Anthropic, Google AI, Perplexity, xAI, Bing, Googlebot.
Deployment ready
Includes grouped rules and sitemap support in one file.

Is your site blocking AI crawlers right now?

We'll read your live robots.txt and tell you which AI crawlers it lets through. No signup, no email.

Configuration Workspace
Adjust crawler access and export a deployment-ready file.
Instant generation
Sitemap URL
Allowed crawlers
Generated robots.txt
Updates instantly as selections change
Preview
robots.txt
# Standard search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# OpenAI / ChatGPT
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Anthropic / Claude
User-agent: ClaudeBot
Allow: /

# Google AI Overviews + Gemini
User-agent: Google-Extended
Allow: /

# xAI / Grok
User-agent: xAI-Grok
Allow: /

# All other crawlers
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml
Notes

ChatGPT Search uses OAI-SearchBot

Blocking GPTBot does not block ChatGPT Search. If `OAI-SearchBot` is blocked, your site can disappear from ChatGPT citations even if classic SEO looks fine.

Google-Extended is separate from Googlebot

A site can be open to Google Search but still blocked from AI Overviews and Gemini if `Google-Extended` is disallowed.

PerplexityBot respects robots.txt closely

If PerplexityBot is blocked, it is unlikely to cite your content. There is no practical workaround beyond allowing the crawler.

Next step

robots.txt is only one of many AEO checks.

Use this file as a starting point, then run the full audit to catch schema, answer structure, content depth, freshness, and other blockers affecting AI visibility.

Run a free AEO audit →
Also useful: llms.txt Generator →