VisibleMax
Get started

Blog category

AI crawlers

robots.txt, GPTBot vs OAI-SearchBot, Claude and Perplexity crawlers, and crawl readiness.

All postsStart free scan

  • AI crawlers · 2026-07-31

    robots.txt for GPTBot vs OAI-SearchBot

    GPTBot is OpenAI’s training crawler. OAI-SearchBot powers ChatGPT search inclusion. Blocking the wrong one quietly removes you from AI answers.

  • AI crawlers · 2026-07-31

    llms.txt explained

    llms.txt is a curated Markdown map of your site for AI agents. It is not a magic ranking signal — but it is useful documentation for tools and RAG pipelines.