How we stopped babysitting CI failures — 3 lessons from routing every failure through one queue
Problem:
CI went red, nobody knew for hours. Then someone dug, someone hand-filed, and someone fixed. The cost was not the failure — it was the queue that did not exist.
We ship an ops queue that delegates failures to coding agents → PR. Supporting infrastructure includes flags, kill switches, and dynamic configs. Core loop: failure → ticket → agent → PR.
What we kept small:
-
Don’t build infrastructure the cloud already gives you. Cloud Build → Pub/Sub filter FAILURE/TIMEOUT/INTERNAL_ERROR → push to /webhooks/cloud_build. No polling.
-
Dedupe at the edge. Pub/Sub and webhooks are at-least-once. Rails.cache.write(key, true, unless_exist: true, expires_in: 7.days) on build or delivery ID means first writer wins and redelivery no-ops.
-
File a real ticket, not a log line. A ticket has a title, reproduction steps, actual and expected result, priority, and tags. Agents can act on that structure. The same shape handles public widget tickets and build failures through one queue and one router.
Setup: npm i -g @shipeasy/cli, then shipeasy setup. It fans out to GitHub, Slack, and an agent-eligible queue.
How are you turning production failures into work your team or agents can pick up without manual triage?
The structured failure queue makes sense. One thing I’m curious about once the coding agent is actually generating the fix: how are you validating that the agent’s response to the failure was the right response, rather than simply that it produced a PR?
For higher-consequence failures, do you sample cases where the agent should have escalated, stopped or required human review?
That distinction between operational automation and independently verifying the resulting behaviour is something we’ve been exploring with OpsWatch.
The "file a real ticket, not a log line" point is the one I'd underline — structured repro steps are exactly what makes a failure delegable, to a teammate or an agent. The hard part I keep hitting is dedupe: one flaky integration test can generate 30 near-identical tickets overnight and drown the queue. Do you fingerprint failures (stack trace hash, test name + error class) before they enter the queue, or does the router handle collapsing? Would also be curious what share of queued failures actually end in a merged PR versus getting closed as flake — that ratio seems like the real proof the loop works.