Your AI passed QA—but is it safe to ship?
AI apps can pass normal tests and still leak data, follow prompt injections, or take unauthorized actions in production.
AI Release Assurance is a pre-release security gate that adversarially tests your staging AI and tells your team whether to ship, block, or review—with evidence explaining why.
Really interesting discussion. The distinction between “the model says it behaved correctly” and having external evidence that proves what actually happened feels especially important.
I also like the idea of treating inconclusive results as review cases rather than automatically passing them. That seems much safer than relying on a single score or the model’s own explanation.
For teams using AI agents with multiple tools, do you think the biggest challenge is detecting the failure itself, or reconstructing enough of the execution path afterward to understand exactly where the control broke?
Exactly. An inconclusive result shouldn’t quietly become a pass. If the evidence can’t show what happened, someone needs to review it.
For multi-tool agents, I think reconstructing the execution path is harder than detecting the failure. Spotting a bad outcome is one thing; understanding which tool ran, what parameters it received, and where the control broke is the real challenge
This is close to the assurance problem we’ve been working on too.
One thing I’m curious about: once a customer moves from testing into an enterprise deployment, are you seeing buyers ask for an independent/client-facing assurance result as well — not just the technical evidence?
We’re finding the gap between “the system passed our testing” and “the enterprise buyer is comfortable signing off” can become a separate problem.
That’s a really good point. We’re making the results client-facing, so buyers can see what was tested, what was found, and why the release was approved or blocked. But we’re not positioning that as independent certification.
I’m curious—what kind of assurance are your buyers asking for before they’ll sign off?
The recurring need is less “certify the AI” and more “give us enough independent evidence to sign off the deployment with confidence.”
The questions tend to be practical:
That’s the gap we’re positioning OpsWatch around: independently comparing expected behaviour with observed behaviour, validating the control path and evidence, then giving a clear verdict with the reasoning visible rather than just producing another score.
I’d be interested in how PromptBrake currently handles that final sign-off boundary.
That makes sense—it sounds like we’re tackling adjacent parts of the same problem.
With AI Release Assurance, customers configure release-gate criteria in their CI workflow—such as allowed failures, warnings, inconclusive results, and minimum test coverage. The PromptBrake runner tests the real release candidate, preserves the evidence, re-tests fixes, and returns an approve, block, or review result.
The customer still owns the final sign-off; we don’t position the result as independent certification. It sounds like OpsWatch adds that independent assurance layer—is it delivered as a service today or built into the product?
Exactly — that’s the distinction I see as well.
OpsWatch is delivered as an independent assurance service today rather than being embedded into the customer’s release tooling.
The customer or their engineering platform can define and enforce the release criteria, run the tests, preserve the evidence and determine whether the release gate has technically passed. OpsWatch then sits outside that control path and independently evaluates whether the observed behaviour, evidence and controls actually support the assurance claim being made.
So in a PromptBrake-style workflow, I’d see the separation roughly as:
PromptBrake → test the real release candidate, enforce configured release criteria, preserve evidence and re-test fixes.
OpsWatch → independently examine the resulting evidence and behaviour, test consequential or boundary conditions where appropriate, verify that the controls behaved as represented, and issue an independent assurance verdict within a defined scope.
Importantly, OpsWatch doesn’t take over the customer’s final production sign-off either. We provide an independent evidence-backed verdict that they can use in that decision.
That separation is why I think the two models are potentially complementary rather than competitive. PromptBrake can create a very strong evidence trail at the release boundary, while OpsWatch can independently verify what that evidence establishes.
A small reference workflow between the two would probably make the boundary very clear.
I can see the distinction. Before exploring a reference workflow, do you have a current client who needs both pre-release testing and an independent review? A concrete deployment would help us understand whether there’s a practical fit.
We do have current deployment opportunities where that combination could be relevant, although I want to be precise: none has yet specified “PromptBrake + OpsWatch” as a formal requirement.
We’re currently involved in live AI deployment and pilot discussions where the customer/vendor needs to establish that the system behaves correctly before production, while OpsWatch’s role is to independently verify the resulting behaviour, controls, evidence and failure paths rather than rely solely on the implementation team’s own testing.
That is actually why I’m interested in exploring the boundary with you.
A practical deployment could look like:
PromptBrake → pre-release testing / release-gate evidence
Deployment → real release candidate
OpsWatch → independent verification of the resulting behaviour and evidence
If we identify one of the current deployments where both layers genuinely add value, I’d much rather use that as the reference workflow than design something hypothetical.
That makes sense, and I appreciate you sharing the context. Our focus right now is making the AI Release Assurance workflow strong at the release boundary—testing the real candidate, applying the gate, and preserving clear evidence.
If a live deployment genuinely needs both layers, I’d be happy to continue the conversation privately and see whether there’s a natural fit.
That sounds like the right boundary.
We’ll keep OpsWatch independently focused on observed production behaviour and resulting system state, while PromptBrake remains authoritative at the release boundary.
We’re currently progressing several production-oriented conversations. If one presents a genuine requirement for both release assurance and independent post-release verification, I’ll bring you the concrete workflow privately—with the deployment context, claims requiring verification and proposed division of responsibility—rather than asking either side to design an integration in the abstract.
I appreciate the openness. I think a real deployment will tell us very quickly whether the two layers fit naturally.
Your strongest line is the first four words, and the rest of the post walks away from it. "AI apps can pass normal tests and still leak data" isn't a feature description, it's the entire reason you get to exist, and it's a claim most teams haven't sat with.
Here's why it lands: normal QA and security testing ask structurally opposite questions. QA asks "does it do what it's supposed to," and you write those tests yourself, so they can only cover behavior you already imagined. Adversarial safety asks "what's the worst thing someone can make it do," which is precisely the space you didn't imagine, that's why it's a vulnerability. No volume of tests you author yourself closes that gap, because you can't test for the attack you didn't think of. That's not a "more testing" problem, it's a "different kind of adversary" problem, and it's why passing QA tells you nothing about safety.
So your wedge isn't "a security gate for AI." It's "your own tests can't catch this class of failure by construction, because you wrote them." Lead with the impossibility, not the gate. The gate is the answer; the impossibility is why anyone needs the answer.
What's the failure a team is most shocked you caught after their AI already passed their own test suite? That specific "it passed our tests and you broke it anyway" moment is your whole demo.
That’s a really useful way to frame it. I’d only soften “your own tests can’t catch it,” because teams can write adversarial tests—they usually just don’t during normal QA.
I don’t want to invent a customer story, but the clearest example we’ve seen is an agent that handles normal booking requests perfectly, then accepts an ID or tool parameter the user shouldn’t control. Everything looks fine until you test the path nobody expected.
That’s the demo moment we’re building around: the normal goal passes, the abnormal path doesn’t—and the evidence shows exactly why.
You're right, and that's a better line than mine — "teams can write these tests, they just don't during normal QA" is the more accurate and more sellable version. I'll concede the correction fully, because it actually sharpens your wedge rather than weakening it.
Here's why your version is stronger. If the gap were "teams can't," you'd be selling a capability, and capabilities get insourced the moment someone smart enough joins. But "teams can, and reliably don't" is a behavior gap, and behavior gaps are durable, because they're cultural, not technical. QA culture is built around confirming intended behavior; adversarial testing requires actively trying to break your own thing, which is a different mindset that doesn't switch on just because it's possible. You're not selling the ability. You're selling the default. And a default only sticks if it's a gate, forced on every release, not a task someone has to remember to do, because the thing people can do but don't is exactly the thing that never happens until it's mandatory.
Your booking example nails why: the failure lives in the path nobody owns. QA owns the happy path, security owns nothing until production, so the abnormal path is an orphan, no team's job until it's an incident. That's the clean framing — you're not adding a test, you're adopting the orphan path, the requests that are nobody's responsibility until they're everybody's emergency.
Is your buyer the QA lead or the security lead? Because "the path nobody owns" is also the deal nobody owns, and which side of that org you enter through probably decides how fast this lands.
“The path nobody owns” is a great way to describe it—and making that path part of every release is exactly the goal.
Our buyer is whoever is responsible for shipping the AI system. That might be a solo developer, founder, engineering lead, QA team, or security team. The title matters less than one question: who has to decide whether this AI release is safe enough to ship?
The "evidence explaining why" part is the piece most gates skip — a bare pass/fail score doesn't tell you what to fix. We ran into something similar building a citation-fidelity checker (every claim in a generated report has to trace back to real source text): a single similarity score let genuinely wrong things through — same structure, wrong entity — because it was never forced to show why it passed. Once we made it expose the actual evidence instead of just a number, we found bugs the score alone had been hiding.
Curious how you scope the "evidence" for injection/unauthorized-action cases — the specific adversarial prompt that got through, or a broader session trace?
Great question. We capture the smallest reproducible trace: the exact adversarial prompt, the response, the failed check, and why it failed. For multi-turn or tool-use cases, we include the conversation leading to the failure and any exposed tool call or parameters. We don’t claim an unauthorized backend action occurred from model text alone—that requires execution evidence.
That distinction — model text describing an action vs. execution evidence that it happened — is an important one, and it's easy to conflate. We hit an analogous issue: a claim can read like it's grounded (right structure, right tone) without the underlying entity/number actually matching the source. Text plausibility and text correctness aren't the same signal, but they're easy to treat as one.
Do you ever see the reverse failure mode — the model claims a guardrail blocked something, but no corresponding tool-call/execution evidence exists (a hallucinated refusal)? Feels like the same trust boundary from the other side.
Yes, that’s the reverse failure mode we watch for. A model saying “I blocked that action” doesn't prove enforcement happened. We only treat it as verified when the integration exposes matching tool-call or execution evidence. If that evidence is unavailable, the result should be inconclusive/review—not a pass. The durable control still belongs server-side; the model’s refusal is only one observed behavior.
Right — same principle either direction: don't trust the model's self-report of what happened, verify against something external. For claims, that's cross-checking against source text; for actions, that's cross-checking against execution evidence.
Funny timing — we just spun off the entity/number matching piece from this conversation into a small standalone free tool (paste an AI claim next to its source, it flags what isn't backed). Same "plausibility ≠ correctness" principle, just for the claims side rather than the actions side. Happy to share the link if it's useful for how you're framing evaluation.
Appreciate you pushing on this from the execution-evidence angle — it's a cleaner way to state the same underlying requirement.
Yeah, send it over—I’d like to try it. Entity and number mismatches are a great example because the output can look completely grounded until you compare it with the source.
I think the shared principle is simple: the evidence has to come from outside the model, not from the model explaining why it thinks it was right. Really appreciate this conversation.
Here it is: https://hallucination-diff-690339828002.asia-northeast1.run.app/en
Well put — "evidence from outside the model" is exactly the right compression of this whole thread. Appreciate you working through it with me too.