Our AI assurance test passed. The problem was that the test was wrong.
This week, we stress-tested part of the evidence pipeline behind OpsWatch, our independent assurance service for production AI systems and agents.
The first result looked encouraging.
We injected 30 distinct items into a context-assembly workflow and tested how many survived the handoff. The test reported that 12 were present.
Then we inspected the evidence more closely.
The assertion was using substring matching.
“Reply 1” was being counted as present when the system contained “Reply 19.”
The assurance test had passed because the test itself could not distinguish between two materially different results.
We replaced substring checks with whole-line set matching, ran the test red first to prove it could detect failure, and added exact item-count assertions.
That exposed the real result.
We then found a second, more consequential problem: provenance survived at the sending boundary but disappeared during summarisation. Both endpoints appeared healthy, yet the receiving agent was operating on a compressed interpretation that had silently lost where the information came from.
This is the assurance gap I think many AI teams are underestimating.
It is not enough to ask:
Did the model produce the expected answer?
Did the control say it blocked the action?
Did both services return successful responses?
Does the log contain something resembling the expected evidence?
You also need to establish:
Did the exact information survive every handoff?
Was provenance preserved?
Did the system execute only what was authorised?
If an action was denied, can you independently confirm non-execution?
Would the verification test itself detect a subtly wrong result?
A control decision is not proof of the resulting system state.
A passing test is not meaningful until you have watched it fail for the right reason.
This is why we are building OpsWatch as an independent layer rather than relying on an agent’s own logs or self-reporting. We examine the actual release candidate or bounded production workflow, preserve the evidence, test consequential failure paths, and return an independently reasoned verdict.
I’m interested in hearing from founders and engineering teams running AI agents in real workflows:
What is the most consequential claim your system currently makes that you have not yet independently verified?
If your agent can change data, communicate with customers, approve something, trigger a workflow, or affect production, I would be especially interested in comparing notes.
And if you have a live deployment approaching release, procurement, or a customer assurance review, feel free to message me. We are opening a small number of bounded, paid verification engagements through McGill Intelligence.
The provenance loss suggests a useful invariant for each handoff: the receiver needs both the compressed content and an independently checkable pointer to the source slice behind every decision-relevant claim.
Otherwise a system can preserve a provenance field while changing the substance it supposedly proves. I would test the handoff with a small mutation: alter one source fact, keep the summary plausible, and verify that the downstream decision is invalidated rather than accepted from stale evidence.
Exactly. A provenance field is only useful if it remains cryptographically or deterministically bound to the source evidence whose meaning it represents.
Your mutation test is the right next step. I would frame the acceptance condition as:
If a decision-relevant source fact changes, every downstream summary, provenance reference and decision derived from that fact must either update consistently or become invalid.
The dangerous failure is not simply losing provenance. It is preserving a plausible-looking pointer while the compressed interpretation no longer matches the referenced source.
I would test three variants:
In each case, the downstream system should reject the evidence, invalidate the decision or escalate for review—not silently continue using the stale interpretation.
That separation between “provenance present” and “provenance still proves the claim” is exactly where independent verification becomes valuable.
The second finding feels more commercially significant than the broken test itself: both endpoints looked healthy while provenance was lost in the handoff.
Curious whether early buyers are asking for broad “AI assurance,” or whether they come with one specific high-consequence workflow they already need independently verified.
So far, the serious conversations begin with one specific, consequential workflow rather than a request for broad “AI assurance.”
The buyer usually has a concrete claim they need to rely on: a verifier blocked the wrong action, an agent respected an approval boundary, a financial workflow escalated correctly, or a release is safe enough to put in front of a customer.
The broader assurance relationship only becomes visible after examining that first workflow. Once you discover that both endpoints can appear healthy while provenance disappears between them, the question expands from “did this test pass?” to “which other handoffs are we currently trusting without independent evidence?”
That is where a bounded verification can become an ongoing assurance requirement. But the commercial entry point is normally one high-consequence claim that somebody already needs proven.
That makes sense. I’ll be interested to see how that first workflow develops as more of these conversations happen.
That is exactly why I’m looking for the next real workflow rather than designing the model in isolation.
Aryan, do you currently have one AI or agentic workflow where an incorrect action, false success signal or broken evidence handoff would create a meaningful business consequence?
If you do, and the workflow already produces inspectable evidence, it may be suitable for a bounded paid OpsWatch engagement. The starting point would be one specific claim you need independently established—not a broad audit of the entire system.
I’d be interested to know which workflow you would choose if you had to put one forward.