A Failed status with crawlability issues means our crawler tried to load your page and could not complete a clean read. This is about whether an automated visitor can reach and load your page. It is not a reflection of your content quality or your AI Optimization Score. When the crawl cannot complete, there is nothing to score, so the run is marked Failed and returns no score rather than a low one.
The good news is that the results give you everything you need to find the cause, and it almost always lives on the website's side: a redirect, a security or bot rule, or a server error.
What is the crawlability issues list telling me?
Each entry in the list is one failed attempt to load your page. Every attempt shows the time it happened, a short description of what went wrong, and, where available, an HTTP status code, a destination address, and how long the server took to respond. Those details are your starting point for tracking down the cause.
A very fast failure usually points to an outright block or redirect. A very slow one that still fails often points to a timeout or an overloaded server.
Why did my audit fail with a suspicious redirect?
This is the most common crawlability failure, and it usually has the simplest fix.
Gumshoe flags a redirect when the URL you entered sends our crawler somewhere else. We treat that as a failure rather than quietly scanning wherever it lands, because the destination may not be the page you meant to measure. If a page on your US site redirects to the UK version, scanning the UK page would hand you UK results under a US label.
Each failed attempt names the address the URL was redirected to, so you can see exactly where it went. That leads to one of two situations:
- The destination is the page you wanted. A home page that redirects to a language or regional version is the usual case, for example a domain that sends every visitor to its /fr version. Add that destination URL to your Technical Audit and it will scan normally.
- The destination is not what you wanted. The URL may be pointing at a login screen, a consent wall, a defunct page, or a different region than the one you are measuring. Audit the correct URL directly, or adjust the redirect rule on your site.
To scan the destination page:
- Open the failed run and find the address listed next to the redirect. That is where the URL actually went.
- If that is the page you want scored, open Technical Audit and click Add pages.
- Enter that destination URL, including any language or regional path and any www prefix, exactly as it appears in the failure detail.
- Click Start page audits.
- Once the new URL runs cleanly, archive the original redirecting URL so it stops failing on future runs.
Small differences count here. A URL with www and one without, or with and without a language path, are treated as separate addresses, so copy the destination exactly rather than retyping it.
What do the status codes tell me?
The HTTP status code next to each attempt tells you how your server, or a service in front of it like a CDN or firewall, responded to our crawler:
- 2xx (success): The server returned a response, but what came back was not usable as a full page. This often means the crawler received a near empty shell or a bot verification or challenge page instead of your real content.
- 3xx (redirect): Your server sent our crawler to a different URL. See the redirect section above, which covers this case in full.
- 4xx (request refused or not found): 401 and 403 mean access was denied, often by a firewall, bot protection, or a login requirement. 404 means nothing was found at that URL. 429 means the server rate limited us for too many requests.
- 5xx (server error): Your server, or a CDN, proxy, or load balancer in front of it, hit an error while responding. 500 is a general server error, and 502, 503, and 504 usually point to a gateway, availability, or timeout problem. These are frequently intermittent.
Why can a page that loads fine for me still fail?
Automated requests are treated differently from a person clicking through in a browser. Firewalls, bot protection, and region based redirects often wave a normal visitor through while blocking or redirecting an automated crawler. That is why a page can look perfectly fine to you and still fail the audit. It is also why results sometimes vary from one run to the next: the protection triggers on some requests and not others.
How do I find and fix the cause?
- Open the failed run from the Analysis results screen and note the exact time, status code, and any destination address for each attempt.
- If the failure is a redirect, follow the redirect section above. That resolves most cases without any changes to your site.
- Otherwise, confirm the page loads publicly in a normal browser.
- Match those timestamps against your own server, CDN, or firewall logs to find the exact requests that failed. The status code tells you where to look: access and bot rules for 4xx, and server or gateway health for 5xx.
- Address the issue on your side (review the security or bot rule, or resolve the server error), then republish the page if you changed anything.
- Return to the Technical Audit, open the page, and click Reanalyze to run a fresh scan.
What if I still need help?
If the page loads cleanly, the URL does not redirect, and your logs do not show a clear cause, ask for a human in chat or email support@gumshoe.ai with the page URL and the date and time of the failed run. We are glad to take a look with you.