When a checker says “all AI crawlers allowed”, it might mean “we could not tell”

The two failures that look identical

When a tool asks your server for /robots.txt, three things can happen. It gets the file. It gets a definitive “that does not exist”. Or it gets nothing usable — a timeout, a connection reset, a 403 from a firewall, a 503 from an overloaded origin.

The first two are answers. The third is not, and the distinction matters because the second and third lead to opposite conclusions. A confirmed 404 genuinely does mean crawlers may fetch anything: with no file, there are no rules, so nothing is disallowed. A timeout means you do not know what the rules are.

The bug is collapsing both into an empty string. Parse an empty string for Disallow rules, find none, and report “allowed”. The logic is locally correct at every step and produces a confident wrong answer.

What the specification actually says

RFC 9309, which standardised the Robots Exclusion Protocol in 2022, treats these cases separately and assigns them opposite meanings.

Section 2.3.1.3 covers “unavailable” — 4xx status codes, including 404 and 410. The crawler may access any resource on the server. No file, no restrictions.

Section 2.3.1.4 covers “unreachable” — 5xx status codes and network errors. The wording is unusually blunt: the robots.txt file is undefined and the crawler MUST assume complete disallow. Not “proceed with caution”. The stricter reading is the required one, and it is the exact inverse of the 404 case.

So a tool that treats a timeout like a missing file has not been slightly conservative. It has inverted a normative requirement.

Why real crawlers are more forgiving than the RFC

Google does not implement the bare MUST, and it is worth knowing the difference before you panic about a 503.

Google's documented behaviour: for the first twelve hours of 5xx responses it stops crawling the site and keeps retrying. For the next thirty days it uses the last known good cached copy. If no cached copy exists, it assumes there are no crawl restrictions. After thirty days, if the site is otherwise healthy, it behaves as though there is no robots.txt at all.

That is a graceful degradation the RFC permits but does not require. It also means a brief outage does not cost you anything, provided your robots.txt was reachable before it and becomes reachable again after.

The practical takeaway is narrower than “a 503 will deindex you”. It is that a persistent 403 — the kind a bot-blocking rule produces, which never resolves and never gets cached — is a different and more durable problem than a transient 500.

The case that actually bites: a firewall that answers everything

A 403 from a WAF is the interesting failure, because it is invisible from a browser. You visit yoursite.com/robots.txt, see the file, and conclude everything is fine. The file is being served to you and refused to anything that looks automated.

This is not hypothetical. Running our own checker against large sites on 13 September 2026, medium.com, quora.com, g2.com and stackoverflow.com all returned 403 to our request. Our user-agent is unremarkable and identifies itself honestly, which is precisely why it gets refused — bot rules key on exactly that.

Worth noting how fast this shifts. nytimes.com, reddit.com and theguardian.com refused us the day before and answer normally now. Any list of blocked sites, including this one, is a snapshot rather than a fact about those companies — which is the argument for testing your own domain instead of reasoning from someone else's results.

Those sites are almost certainly reachable by OAI-SearchBot and PerplexityBot, because major AI crawlers publish IP ranges and get allowlisted by CDNs. So the honest conclusion for a large site is “we cannot see it, but the real crawlers probably can”. The dangerous case is the mirror image: a small site behind an aggressive default WAF rule, where nobody has allowlisted anything, and where “our checker was refused” is a genuine warning rather than a false alarm.

How to check it yourself in one command

Skip the browser — it is the one client guaranteed to get a friendly answer. Use curl with a user-agent that a bot rule would recognise, and look at the status code rather than the body. The -L flag matters: plenty of domains redirect /robots.txt, and without it you are reading the 301 rather than the file.

Run this, substituting your domain:

curl -sL -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yoursite.com/robots.txt

A 200 means the file is being served to something that announces itself as a crawler. A 403 or 503 means it is not, and that is worth investigating regardless of what any checker told you. Compare it against the same request with a browser user-agent — if the two differ, you have a bot rule deciding who sees your crawl directives.

One trap worth avoiding: do not add -I to send a HEAD request instead. Testing this, quora.com returned 403 to HEAD and 200 to GET for the same URL and user-agent. Some bot rules treat HEAD as more suspicious than GET, so a HEAD-based test manufactures exactly the false alarm this article is about. Crawlers fetch with GET, so test with GET.

For OpenAI specifically, verification can go further: OpenAI publishes per-bot IP ranges at openai.com/searchbot.json, gptbot.json and chatgpt-user.json. If your firewall logs show blocks from those ranges, the problem is confirmed rather than inferred.

What our own checker does, and where it departs from the spec

Our AI visibility checker returns one of three states for robots.txt: read successfully, confirmed missing, or unreadable. A confirmed 404 scores full marks, because per the RFC that genuinely does mean unrestricted. An unreadable robots.txt scores partial marks and is reported as unverified, with the check marked as failed rather than passed.

We should be precise about one thing, because this article is about other people's imprecision. The RFC says a crawler MUST assume complete disallow when robots.txt is unreachable. We do not report a score of zero. A diagnostic tool and a crawler have different jobs: a crawler has to decide whether to fetch, so it needs a default, and the safe default is to stay out. A tool has to tell you what is true, and what is true is that the state is unknown. Reporting zero would trade one confident wrong answer for another.

This was a real bug in our own code, not a hypothetical we identified in others. Our first version compressed “fetch failed” into an empty string and awarded a perfect crawler-access score to nytimes.com — which was refusing us at the time — reporting all ten AI crawlers as allowed when we had read nothing at all. We found it by running the tool against sites we did not control, which is the only reliable way to find this class of bug.

How to evaluate any checker on this

You do not need to read anyone's source code. Point the tool at a domain whose robots.txt is known to refuse automated requests — the publishers listed above are convenient — and read what it claims.

If it reports a specific number of crawlers allowed, it is reporting a parse of nothing. If it distinguishes “no robots.txt found” from “could not read robots.txt”, it is handling the case. If it says nothing about robots.txt at all, you have learned less than the silence suggests.

This generalises past robots.txt. Any check that can fail to gather its input needs three outcomes rather than two, and the third one is where tools reveal whether they were built to inform you or to produce a reassuring green tick.