Measuring AI visibility: methods, metrics and pitfalls

Why a single question proves nothing

Anyone who wants to know whether an AI recommends their company first types in a question and looks at what comes out. That is understandable, but it is misleading. AI answers fluctuate: the same question, asked twice, often produces two different lists of names. Sometimes you are in it, sometimes not. A single answer is therefore more of a coin toss than a measurement.

On top of that: the one question that happens to occur to you is rarely the one your customers ask. You think in terms of your offering, your customers think in terms of their problem. Someone looking for a hotel does not ask for the hotel name, but for "quiet, with sauna, by the lake, dog-friendly". A serious measurement therefore needs many questions and many runs, not one lucky hit.

The question panel: the heart of every measurement

The most important building block is a fixed question panel: a collection of a hundred to several hundred questions that covers your customers' entire decision journey. From the first, still vague problem ("Where can I find...") through the comparison ("Which is better, A or B?") to the concrete question about your name. Each of these stages measures something different.

The panel is built carefully once and then stays unchanged. This is not convenience but the basic condition for comparability: only someone who asks exactly the same questions every month may lay the results of two months side by side. If you change the panel, from that point on you are measuring something else and you lose the time series.

A good panel also reflects the language of customers, not that of marketing. It contains typos, colloquialisms, regional terms and the small additions that make a question real. The closer the panel is to real demand, the more meaningful the score.

Several AIs, several runs

There is no single AI. ChatGPT, Claude, Gemini and Perplexity draw on different sources and answer differently. Anyone who measures only one sees only a slice of their market. A robust measurement therefore queries several assistants in parallel and keeps the results separate, so you can see where you are strong and where you are blind.

And because the answers fluctuate, each question is asked multiple times, typically three to five times. Only the average over several runs is a value you can build on. A measurement that asks each question just once confuses chance with result.

The three metrics that matter

From the ratings three metrics can be derived that together give an honest picture. The first is the visibility score: a number from 0 to 100 that condenses how present you are overall. Being mentioned counts, being recommended counts more, standing in first place counts most.

The second is the share of recommendations, often called share of voice. If a hundred questions produce three hundred names mentioned and thirty of them are yours, your share is ten percent. This number shows how much of the pie you get and who eats the rest.

The third is the breakdown by question type. A business can shine on questions about its own name and be completely absent from the category questions through which new customers arrive. The overall score alone hides this; the breakdown makes it visible and shows where work has the greatest leverage.

The most common pitfalls

The first mistake is the wandering question. You change the panel because a better wording occurs to you, and in doing so you destroy comparability. Rule: improvements go into a separate new panel, the old one stays for the time series.

The second mistake is the one-time snapshot declared to be the truth. Without repetition you are measuring noise. The third mistake is checking only your own brand and ignoring competitors; then you see your own standing but not the market. The fourth mistake is confusing the score with revenue: it measures visibility, not bookings. It is a leading indicator, not a receipt.

• Changing the panel over time and then comparing months
• Asking each question only once and taking the result at face value
• Measuring only one AI and overlooking the rest of the market
• Not measuring competitors as well
• Reading the score as a revenue forecast

Measure yourself or have it measured?

You can get a rough feel yourself: ask your ten most important customer questions to two or three AIs, each three times, and note whether you appear. That does not replace a measurement, but it shows you within half an hour whether you have a problem. For anything beyond that, method pays off: a fixed panel, several AIs, several runs, documented answers and a time series over months.

The real value of the measurement lies not in the number but in the to-do list it produces. Every question where you are missing tells you exactly which answer does not yet exist on the web. That turns the diagnosis into a work plan, and the score into a tool rather than a vanity metric.

What is a good visibility score?

The question about the "good" value cannot be answered with a single number, because it depends on the market. In a fiercely contested field with many well-known providers, a score of 30 is already strong, while in a niche with little competition it would be rather weak. The score only gains meaning through comparison: against your own previous months and against the competitors in the same panel.

More telling than the absolute value is therefore the movement. A business that climbs from 9 to 40 in six months has achieved more than one that sits steadily at 45 without moving. The first builds a lead, the second merely manages it. Anyone reading the score should always ask two questions: where do I stand compared with those who win the same questions, and in which direction does my curve point?

Just as important is the distribution behind the score. Two businesses with the same value of 35 can stand in completely different positions: one is a little visible everywhere, the other is right at the front on a few questions and invisible otherwise. For the next measure this difference is decisive, and only the breakdown makes it visible.

Manual sample or systematic measurement

Between the quick self-test and the full measurement there is a big difference in effort, but also in benefit. The manual sample is done in half an hour and answers a single question: do I even have a problem? For that it suffices. For everything that comes after, it is too coarse, because it neither smooths out the fluctuation nor captures the development over time.

A systematic measurement takes this work off your hands and makes it comparable. It asks the same panel every month, documents every answer verbatim and builds a time series from which cause and effect can be read. The real gain is not the automation but the discipline: because the conditions stay constant, you actually measure progress and not the weather of the day.

For most companies the sensible sequence is therefore clear: first the manual sample to decide whether the topic is worthwhile, and then, if the answer is yes, the systematic measurement as the basis for the actual work.

Why competitors belong in every measurement

Measuring your own visibility alone is like timing your own run without knowing how fast the others are. A number without comparison says little. Only when the competitors are measured in the same panel does a picture of the market emerge: who is recommended on which questions, who shows up everywhere, and where is the field surprisingly empty?

This view changes priorities. Questions where a strong competitor dominates the answer are hard to win and should wait. Questions where no one is really visible, by contrast, are open doors: there a good answer page is often enough to be the first to take the spot. Without competitor data you overlook these opportunities and spread your energy evenly instead of deliberately.

In practical terms this means: from the outset include three to six relevant competitors in the panel and measure them with the same questions. The comparison costs almost no extra effort, but it turns the score from an isolated number into a map on which you can see where the next step pays off most.