Five questions to ask an AI visibility tool before you trust its score

Why two tools give you two different answers

The scores disagree because the underlying question is harder than it looks. “Are you visible in AI search” has no single measurable value the way “what position do you rank at” does. It depends on which prompts you ask, which engine answers, whether that engine searched the web on that particular call, and when you asked.

Change any one of those and the answer changes legitimately. So a score is only meaningful alongside the choices that produced it, and most tools present the score without the choices.

None of what follows requires you to distrust these tools. It requires you to know which question a given number is answering, because buying the wrong one is expensive and the difference is not visible from the pricing page.

Question 1: does it actually run a search, or just ask the model?

This is the one that invalidates entire datasets, and it is the least visible from outside.

If a tool queries a model through a developer API with web search disabled, the model answers from training data. It will still produce confident brand names and citations. Those citations are recalled or reconstructed rather than retrieved, which means they can point at pages that no longer exist, or never did.

Promptwatch published a comparison that illustrates the failure mode well, though its framing needs one correction. They ran commercial prompts through the ChatGPT interface and through the API on the same day, collecting 562 citations, and found only 25.6% overlap. Of the API-sourced citations, 87% pointed at bare homepages rather than specific pages, and one recommended a company that does not exist.

The correction matters: web search was off on their API run. So the study demonstrates that a model answering without retrieval produces unusable visibility data — not that APIs are inherently unreliable. An API call with search enabled is a different thing entirely. Promptwatch sells interface-based tracking, which is worth knowing when reading their framing, and their own glossary hedges more carefully than the headline does.

What to ask a vendor: was retrieval enabled on every call, and can you show me an answer where the model searched? If they cannot distinguish retrieved citations from recalled ones, the citation counts are not measurements.

Question 2: how many times did it ask?

Ask an AI engine the same question five times and you will get five different brand lists and five different sets of sources. This is not a defect to be engineered away; it is how the systems work.

The consequence is arithmetic. A visibility score built on one query per prompt per week is a single sample of a distribution. It will move between runs for reasons that have nothing to do with your website, and you will be tempted to explain those moves in terms of changes you made.

Sampling is the one lever that genuinely fixes this, and it is the one thing almost nobody publishes. Of the tools we examined, achtung.app is the only one that states its sampling design: two runs per query per engine per day, averaged, on its entry tier, and three runs per keyword on the weekly tier — trading frequency for repetition. Two runs is statistically thin. It is also more than we could determine about any competitor, several of which charge considerably more.

The others publish frequency only. llmmetrix states daily checks on paid plans across up to seven models and weekly on free. citlyze states weekly for AI Overviews and daily for answer engines. amicited describes a fixed daily schedule from clean sessions with full answers stored. Frequency tells you how often a coin was flipped, not how many times per flip.

Question 3: can you click through to the raw answer?

This is the fastest way to judge a tool, and it needs no technical knowledge. Take any aggregate number — mentions, share of voice, citation count — and try to reach the actual text the model produced.

If you can, you can verify the number yourself, spot the cases where your brand was mentioned dismissively, and notice when a “competitor” in your report is a company that does not exist. If you cannot, you are being asked to trust a summary of data you are not permitted to inspect.

Storing raw answers costs real money, which is why not every tool does it. It is also the single feature that separates a measurement from an assertion.

Question 4: what does it admit it cannot measure?

Every tool in this category has gaps, because parts of the surface have no automated measurement path. Google AI Mode, in-app advertising, and anything requiring a logged-in personalised session are all genuinely hard or impossible to sample at scale.

So the presence of a limitations section is informative in itself. achtung.app publishes one and introduces it by saying the list is intentionally public because it is the most honest answer to where the product stands — then lists no revenue attribution, incomplete hallucination detection, and per-engine caveats including that Gemini decides per call whether to search, which cannot be forced.

We could not find comparable disclosure from the three commercial tools we looked at. That is not evidence they measure more; it is evidence they discuss it less. When a vendor charging hundreds a month has no published limitations, the reasonable inference is that the limitations are undisclosed rather than absent.

Question 5: is this measuring access, or measuring mentions?

These are two different products and the distinction decides which one you need.

Access checks ask whether AI engines can reach and parse your content: does robots.txt admit the crawlers that decide answer inclusion, is your text in the HTML or only after hydration, is there structured data. These are deterministic. The same page checked twice gives the same answer, and a free instant check is a complete answer rather than a teaser.

Mention tracking asks whether engines actually recommend you when buyers ask. That requires running prompts repeatedly over time against several engines, which costs money per call. It cannot be free and it cannot be instant, and that is the honest reason this category has a paid tier.

Being direct about our own position: the tools on this site are the first kind. They are free because deterministic HTML parsing has no per-check cost, and they tell you nothing about whether ChatGPT recommends you over a competitor. If that is the question you have, the answer is a paid subscription somewhere — ask it the five questions above first.

The order to do this in

Access before mentions. If robots.txt blocks OAI-SearchBot, no amount of prompt tracking will tell you anything useful, and the tracking subscription will faithfully report that you are invisible for a reason a one-minute free check would have found.

Fix access first, confirm it, then decide whether continuous mention tracking is worth paying for. That order costs nothing and eliminates the most common cause of zero AI visibility before you start paying to measure it.