How much can your AI answer change before it's a bug?
I'm building iPulse AI, an Open Agentic Investment Research Platform, and there's a small reliability test I'd like to run: ask the same question twenty times to the same web-search enabled agent , using exactly the same documents and settings.
Different wording wouldn't worry me. Different revenue figures would. The company didn't change its accounts because I clicked again.
I'd start with a narrow question: what changed in this company's cash flow, and what drove the change? Freeze the documents, model version and date. Run each attempt in a fresh session. Record the figures, sources and main explanation, not just the finished paragraph.
Then separate two problems. Did the system pull different evidence? Or did it reach different conclusions from the same evidence? Those need different fixes. I'd also check the answers against the source, because being wrong twenty times is technically consistent.
I wouldn't demand identical judgment on a genuinely uncertain question. But if the conclusion keeps changing, I'd want that uncertainty visible rather than let one lucky run become the demo.
I haven't run this test yet. For people building AI products: which parts of your answer must stay the same when the inputs haven't changed?
Your test setup is stronger because it fixes documents, model version, and date instead of treating nondeterminism as an abstract number. I’d separate retrieval drift from reasoning drift by logging the exact passages and a normalized set of extracted figures for every run. For a cash-flow question, an assertion like “same period, same currency, same source rows” can turn a vague reliability test into a regression check.
For reliability tests I’d log retrieved passages + source versions separately from model reasoning — same-session retries and explicit abstentions included. That way you can tell “retrieval drifted” from “the model just phrased it differently,” which matters a lot when the docs are frozen.
The evidence-vs-reasoning split is useful. At Worldesk.ai , we treat any run-to-run drift in FX verification as a bug, the rate can move, but the source and method shouldn’t.
Same job card across tools helps more than chasing a “best” model. Keep blanks, stops, and artifacts identical when you move Grok Bot ↔ Claude ↔ ChatGPT — then change only the paste target. That usually cuts the “is this a bug or drift?” loop.
I’d define the failure boundary before looking at the outputs, otherwise the evaluator can quietly become “did this run feel different?” A practical split is: factual invariants (figures, dates, cited evidence), decision invariants (the recommended action and its confidence), and acceptable explanation variance. For each, set a tolerance and log the exact retrieved passages plus model/version. I’d also include a known-answer fixture and an abstention case—an agent that reliably says “insufficient evidence” may be healthier than one that produces a confident but unstable investment conclusion. That gives you regression signals you can act on, not just a variance percentage.
You've designed this well, so here's what running it taught me. I ran the same shape against a runtime governance layer this week: one request, frozen inputs, repeated runs, every trace kept. Different domain, same structure.
Two things I'd add before you start.
Run the same-session retry as its own condition, not just fresh sessions. That's where my flips lived. Twenty runs, fresh session each: 20/20 identical. Same session, same request again one second later: 6/6 flipped, decision class and severity both. Nothing in the inputs changed. The fact that the system had seen the request before did. If I'd only tested fresh sessions I'd have called it deterministic and been wrong, and real users retry the moment they dislike the first answer.
Then score the decision and the routing separately. In my runs the request understanding was byte-identical every time; what moved was escalate vs clarify and the severity that routes it. For you that's figures, sources, conclusion, and recommended action, each with its own repeat rate instead of one score.
On your question: figures, sources, and the recommended action should hold under frozen inputs. Wording can vary. If the action can flip between runs, that's the bug, and I'd surface the flip in the receipt rather than hope it doesn't happen.
Raw record from that batch, all trace ids, if useful:
https://pub-a941bfd863a24f91a60e6c4979c18a84.r2.dev/pi-sandbox-uploads/349923636603260928/2026-09-09/1788931281989-388dff5b-9b2b-4885-b75b-3328acc9b1e4-nees_dist_record.json
The distinction between evidence drift and conclusion drift seems critical for an investment product. When you run the test, which one would make users trust the system less: different numbers/sources, or the same evidence producing materially different investment conclusions?
I would separate factual invariants from interpretive variance. Revenue figures, dates, and cited source passages should remain stable when the inputs are frozen. The explanation can vary, but it should not quietly reverse the conclusion without identifying the uncertainty that caused it.
It may also be useful to test whether twenty runs produce the same recommended action, since users often care more about whether the system changes their decision than whether the wording changes.