Same numbers, wrong company: the AI citation bug nobody's checking for

Last week I wrote about two companies named BrandScope — and watched an AI hand one's credibility to the other. Several people asked the natural follow-up: once the AI gets the name right, are we good?

The uncomfortable answer is no. And how I found out is worth sharing, because it happened in a public thread with someone who builds citation-checking software.

We used to bucket citations into three: no source, misleading, reliable. He made me see a fourth.

I asked him a question that sounded naive: "When a citation points to a sentence that doesn't exist — not a paraphrase, not a swap, just missing — how do you route that?"

His answer reframed the problem: "doesn't exist" and "exists but wrong" are different failure modes with different fixes. Conflating them, he said, made his error messages useless for a while.

His routing:

The citation doesn't resolve to any real text → no_basis. There's nothing to compare, so you don't even run similarity.
It resolves and the content matches → pass.
It resolves but entities/numbers don't line up → escalate.
The "why" is the useful part: a missing citation is a retrieval/indexing problem (a hallucinated paragraph number, a phantom sentence); a wrong-entity citation is a semantic-matching problem. Put them in one bucket and your fix tries to solve two unrelated things at once.

The sharpest trap he shared: same numbers, wrong company.

Sentence structure is nearly identical — so a naive similarity score can rate a wrong-entity citation as more supported than a straight fabrication. The model came with receipts. Just not yours.

Their fix: an explicit proper-noun/number check that overrides the similarity score when they don't match. And this week he confirmed the same bug class shows up with role titles and product names too, not just company names. It generalizes — which is exactly why a text-similarity check alone will never be enough.

For us this collapses into one line: mentioned ≠ attributed ≠ evidence-backed.

Coverage says you exist. Attribution says the thing being said about you is actually about you. Evidence says the source behind the claim genuinely supports it. Most brand monitoring stops at coverage — "we appear in 4 of 6 engines!" — and the expensive failures live in the last two layers.

You can check the third layer by hand in 30 seconds when an AI answer actually matters: does the cited URL resolve? Who owns the source — you, or a namesake? Does the source sentence really say what the answer claims? If any answer is no, treat that citation as noise.

We built BrandScope to run all three layers automatically for brands — mentions, attribution, and the evidence behind every citation, across 6 AI engines, every week. That's the whole product. → brandscope.dev

Question for you: when an AI answer comes with citations and you're about to act on it — do you verify the source, or trust the citation? Curious where other founders land.