AI crawler

An AI crawler is an automated user-agent operated by an AI company (OpenAI, Anthropic, Perplexity, Google, Meta, ByteDance) that fetches web pages either to train models or to retrieve real-time content for answer generation. Most AI crawlers are invisible to Google Analytics and other client-side analytics because they don't execute JavaScript.

The major AI crawler families in 2026 split into two broad categories. Training crawlers — GPTBot, ClaudeBot, Google-Extended, Bytespider — fetch pages to add to model training corpora. Live-retrieval crawlers — OAI-SearchBot, ChatGPT-User, PerplexityBot, Meta-ExternalAgent — fetch pages in real time when a user query needs current information.

Each crawler has a published user-agent string and a documented IP range (or ASN). Respecting them means: (a) allowing the user-agent in robots.txt if you want the model to be able to cite you, (b) serving the same content to the bot as you do to humans (no cloaking), and (c) ensuring the page renders in static HTML or has a sensible no-JS fallback (most AI crawlers don't execute JS).

Blocking AI crawlers is increasingly common — especially the training crawlers — for publishers concerned about content reuse. The trade-off is direct: blocking GPTBot in robots.txt removes the page from ChatGPT's training set, but blocking OAI-SearchBot or ChatGPT-User removes the page from real-time citations, which is usually the opposite of what marketing teams want.

AEOlens AI Agent Analytics tracks 8 crawler families server-side (the only reliable way to see them, since they're invisible to JavaScript-based analytics), verifies authenticity by user-agent + ASN, and shows which pages each crawler is fetching and how often.