Website unreachable

If the crawler cannot fetch your homepage and discovered URLs, the scan fails at crawl — before analyze or probe. Unreachable sites are the most common hard failure.

Symptoms

• Scan fails quickly with connection, DNS, SSL, or timeout language
• pages_crawled stays at 0 or very low
• Technical audit empty or missing
• Site works for you but not for datacenter/crawler IPs (bot blocking)

Reachability checklist

1. 1

   Open the exact project URL

   Use a private window. Confirm HTTPS loads without certificate warnings, interstitial challenges, or infinite redirects.

2. 2

   Validate DNS

   Ensure A/AAAA or CNAME records point at the live host. Fresh domains and mis-cutovers often return NXDOMAIN to the crawler.

3. 3

   Fix SSL

   Renew expired certificates, complete chain installs, and align the cert hostname with the project URL (including www vs apex).

4. 4

   Relax bot / WAF rules

   Cloudflare Bot Fight Mode, aggressive WAFs, geo blocks, and IP reputation rules can block crawlers. Temporarily allowlist or lower sensitivity for the marketing site during scans.

5. 5

   Remove auth walls from the crawl entry

   Point the project at a public marketing URL, not an app login. Basic-auth or SSO gates prevent crawl.

6. 6

   Reduce timeout pressure

   Slow TTFB, heavy client-only apps, or huge HTML can hit fetch timeouts. Improve server response or expose server-rendered content — see Slow scans.

Edge cases

• Staging behind VPN — VisibleMax cannot reach private networks; use a public staging host or production marketing site.
• Geographic restrictions — if the origin blocks non-local IPs, crawls from other regions fail.
• Mixed content / HTTPS redirects — HTTP project URLs that bounce oddly can confuse discovery; prefer canonical HTTPS.
• Parked or under-construction pages — may crawl but yield thin evidence (see Missing results).
• robots.txt disallow — may limit discovery even when the homepage loads; review crawl rules if page counts stay thin.

When to contact support

• Public HTTPS works worldwide (e.g. via public checkers) but VisibleMax still cannot crawl
• You need help interpreting WAF logs vs. VisibleMax scan timestamps
• SSL appears valid to browsers but the crawler reports certificate errors

FAQ