Technical audit

Crawl signals, robots (GPTBot vs OAI-SearchBot), llms.txt, schema, and entity clarity checks.

Updated 2026-08-06

The technical audit scores infrastructure readiness for AI crawlers and machine-readable understanding. It runs alongside content analysis and reports issues by severity (critical, warning, info) with fix guidance.

Audit categories

Category	What we check
`robots`	Presence and rules for AI user-agents — retrieval vs training bots
`llms`	Whether `/llms.txt` exists and points to high-value pages
`schema`	JSON-LD Organization / Product / FAQ and consistency
`meta`	Title, description, and basic meta hygiene
`canonical`	Canonical URLs that avoid duplicate entity confusion
`sitemap`	Sitemap discoverability for crawl prioritization
`rendering`	Whether key content is available to crawlers (not only client-only shells)

GPTBot vs OAI-SearchBot (and peers)

OpenAI (and other vendors) split retrieval crawlers from training crawlers. Blocking the wrong one quietly removes you from live AI answers.

Common AI crawler roles
User-agent	Role	Guidance
OAI-SearchBot	ChatGPT search indexing	Allow if you want ChatGPT search citations
ChatGPT-User	Live user-triggered fetches	Allow for on-demand retrieval
GPTBot	OpenAI training	Allow when you want content in training/grounding; decide separately from search
Claude-SearchBot / Claude-User	Claude search / live fetch	Allow for Claude search visibility
ClaudeBot	Anthropic training	Separate from Claude-SearchBot
PerplexityBot / Perplexity-User	Perplexity retrieval	Allow for Perplexity citations
Google-Extended	Gemini / AI grounding uses	Not the same as Googlebot; decide intentionally

llms.txt

llms.txt is a Markdown map at your site root that summarizes the company and key URLs for AI systems. It complements — it does not replace — clear HTML pages and crawler access. VisibleMax audits for it and can generate a starter via Content Studio (llms_txt asset type).

# Acme Analytics

> B2B product analytics for growth teams

## Key pages
- [Home](https://example.com/)
- [Docs](https://example.com/docs)
- [Pricing](https://example.com/pricing)
- [FAQ](https://example.com/faq)
- [Compare](https://example.com/compare)

Minimal llms.txt shape (illustrative)

Schema and entity clarity

Publish Organization (+ sameAs profiles) and Product JSON-LD that match visible HTML facts.
Keep brand name, category, and pricing claims consistent across About, homepage, and docs.
Add FAQ schema when you ship FAQ pages — assistants retrieve structured Q&A aggressively.
Use Studio’s schema and trust assets to draft machine-readable and trust-signal improvements together.

Fix flow

1
Read severity first
Ship critical robots/rendering issues before polishing meta tags.
2
Generate a fix pack
From Studio, generate llms_txt, schema, and related technical drafts; download or copy into your repo.
3
Deploy and rescan
Confirm /robots.txt and /llms.txt return 200, then rescan (Pro) or wait for weekly monitoring.
4
Re-check the technical score
Passed checks should rise; remaining warnings can enter the next weekly plan.

FAQ

My site blocks all bots for security — can I still use VisibleMax?

Scans may fail or return thin crawls. Allowlist VisibleMax crawl infrastructure and AI retrieval bots on public marketing/docs hosts. See Website unreachable.

Does allowing GPTBot put private app data at risk?

Only crawl what you publish publicly. Keep authenticated app routes disallowed. Training/retrieval bots should only see the public marketing and docs surface.

Is the technical score the same as the AI Visibility Score?

No. Technical score is infrastructure checks. AI Visibility Score is broader structural readiness including content and entity clarity. See AI Visibility Score.

Still stuck? Contact support with your project URL and what you already tried.