The separation between implementation and assurance is interesting.
I’d be curious what usually creates the need for that second layer in practice: a specific failure, an external requirement, or simply lack of confidence in the system’s behaviour.
The independence angle is the key part that most teams miss. We're currently relying on a mix of unit tests, evals, and manual spot-checks before releases, but once an agent is live the operational evidence (logs, human overrides, tool call traces) is rarely reviewed by someone other than the builder. A few questions: do you pull evidence from existing observability stacks (e.g., OpenTelemetry traces, LLM provider logs), and how do you define "agreed expectations" for a workflow that changes frequently?
We don’t require teams to replace their existing observability stack. Where appropriate, OpsWatch can work from evidence already being generated — for example traces, provider logs, tool-call records, override events, escalation records and other workflow telemetry — provided it’s sufficient to support the assurance scope.
On “agreed expectations,” we define them at the workflow level before review: intended outcome, permitted actions, boundaries, escalation/fallback behaviour, material failure conditions and any client or operational constraints. If the workflow changes materially, the assurance outcome should be treated as version-specific and re-verified rather than assumed to carry forward indefinitely.
The idea is to make the assurance layer practical enough to sit on top of existing production evidence, while remaining independent from the team that built the system.
About
OpsWatch exists because once AI moves from a demo into real operations, the question changes from “does it work?” to “can we independently prove it is behaving as intended?”
Most teams already have developers, monitorin
3 Comments
The separation between implementation and assurance is interesting.
I’d be curious what usually creates the need for that second layer in practice: a specific failure, an external requirement, or simply lack of confidence in the system’s behaviour.
Yes — that’s exactly the gap we’re focused on.
We don’t require teams to replace their existing observability stack. Where appropriate, OpsWatch can work from evidence already being generated — for example traces, provider logs, tool-call records, override events, escalation records and other workflow telemetry — provided it’s sufficient to support the assurance scope.
On “agreed expectations,” we define them at the workflow level before review: intended outcome, permitted actions, boundaries, escalation/fallback behaviour, material failure conditions and any client or operational constraints. If the workflow changes materially, the assurance outcome should be treated as version-specific and re-verified rather than assumed to carry forward indefinitely.
The idea is to make the assurance layer practical enough to sit on top of existing production evidence, while remaining independent from the team that built the system.