The dashboard was green. Repeat runs weren’t.

Shipping LLM changes is stressful for one reason: “looks fine” isn’t evidence.
A few spot checks and a green dashboard don’t answer the real question: is this specific change safe to deploy?

So we started gating releases on saved real cases:

Same inputs (saved snapshots)
Same checks
Repeat runs = 10×
Only change: system prompt (baseline vs candidate)
Decide go/no-go from case-level verdicts (HEALTHY / FLAKY / FLAGGED), not averages
What changed (candidate vs baseline):

Baseline system prompt (in demo): a normal SaaS support bot prompt (billing/refunds/account/security + “don’t ask for secrets”).
Candidate system prompt (Release Gate override): for refunds/charges, don’t provide instructions; ask the user to contact support, but still summarize what you understood in one sentence; keep replies under 35 words.
Repro/demo code: support-bot-regression-demo

The screenshot shows why this helped us: under repeats, some cases stay stable (10/10), while others become flaky (e.g., 9/10, 8/10, 6/10 healthy — in this run it’s mostly latency).

Question: what’s your minimum pre-deploy evidence for LLM changes — case count, repeat count, and ship-stoppers?

(Optional last line, 1 sentence) We run this loop in PluvianAI (capture → saved cases → pre-deploy replay), but I’m mainly trying to learn what routines other teams use.
Repro (demo code + questions set): https://github.com/JinBongJun/support-bot-regression-demo
Tool we used for capture + replay + gate: https://www.pluvianai.com/