Blog
robots.txt for GPTBot vs OAI-SearchBot
GPTBot is OpenAI’s training crawler. OAI-SearchBot powers ChatGPT search inclusion. Blocking the wrong one quietly removes you from AI answers.
GPTBot collects content for OpenAI training. OAI-SearchBot indexes sites for ChatGPT search. To appear in ChatGPT answers, allow OAI-SearchBot. Decide separately whether to allow GPTBot for training. The same split exists for Claude (ClaudeBot vs Claude-SearchBot) and Perplexity (PerplexityBot / Perplexity-User).
Recommended default for brands that want AI visibility
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the related *-User fetchers. Allow Googlebot for Google Search and AI features that rely on Search indexing.
Allow GPTBot, ClaudeBot, and Google-Extended when you want content available for training and grounding — VisibleMax’s default robots template allows both retrieval and training crawlers.
Common mistake
Blocking all “AI bots” in a WAF or robots file often blocks retrieval crawlers. That removes you from live AI citations while traditional SEO may still look fine.
Validate with VisibleMax’s free robots.txt tool at /tools/robots-txt.
Next steps
Run a free scan on the VisibleMax homepage, browse the glossary, or read more on the blog.
Published 2026-07-31