AI Crawlers

Automated bots deployed by AI companies to index web content for model training and real-time retrieval, including GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity). Managing which crawlers can access your site via robots.txt is a practical AEO consideration: blocking training bots protects your content from being absorbed without attribution, while allowing retrieval bots ensures you can still appear in AI-generated answers.