Nobody Read the AI's Output. They Just Approved It.
The review process didn't disappear when AI entered the workflow. It became theater.
Last week, Wiz Research published a post-mortem on a vulnerability they found in Snowflake's GitHub. The details are technical, but the relevant part isn't.
A pull request was submitted. GitHub Copilot co-authored part of the change and marked it clean. GitHub's Advanced Security scanner analyzed the final revision and flagged nothing. A human engineer approved the merge.
Five days later, an autonomous AI security agent found the injection vulnerability, exploited it, and exfiltrated a Jira token with access to Snowflake's internal engineering, security compliance, and bug bounty projects.
The PR had been reviewed. By AI tools. By a human. Nobody caught it.
The process was followed. The process produced the wrong result. And nobody in the approval chain had actually read what they were approving.
This isn't a story about Snowflake's security practices. It's a story about what "review" means when AI is producing the output being reviewed.
We're running into this in every B2B deployment we do now. Not in CI/CD pipelines — in business processes. The AI drafts the customer communication. A human approves it. The AI routes the intake form. A coordinator signs off. The AI generates the contract clause. Legal marks it reviewed.
The review step is still there. The human is still in the loop — technically. But the review has become something different: it's checking that the format is correct, that the output looks reasonable, that nothing is obviously wrong.
It is not reading the output the way you'd read something a junior employee submitted for the first time.
When AI produces something that looks authoritative, humans apply a lighter hand than when a human produces something that might be wrong.
The real estate staging deployment: where we first noticed the pattern.
We worked with a real estate staging company — residential and commercial, mid-market, high volume — that had deployed an AI system to generate property descriptions and client-facing staging recommendations. A human coordinator reviewed each output before it went to the client.
Three months in, the client's operations lead flagged something. A staging recommendation had gone out with measurements for a room that didn't exist in the property layout. The AI had hallucinated a dimension. The coordinator had approved it.
We pulled the review logs. The coordinator was processing roughly 40 outputs per shift. Average review time per output: 23 seconds.
Nobody had designed the workflow expecting a human to catch a hallucinated room measurement in 23 seconds. Everyone had designed the workflow with a human in it, which felt like the same thing.
"I thought the AI was checking itself," the coordinator told us. "It looked right. I was checking for typos."
The human hadn't been removed from the process. The human's role had been quietly redefined — from reviewer to formatter — without anyone making that decision explicitly.
The film production deployment: what "AI-assisted" actually means in practice.
A film production company we worked with used an AI system to draft initial budget breakdowns for production proposals. The workflow: AI generates draft, line producer reviews, CFO approves, proposal goes to the client.
The line producer was a 20-year industry veteran. She knew production budgets the way a surgeon knows anatomy. When she reviewed an AI-generated breakdown, she caught errors other people wouldn't see.
We ran a usage audit at month four. In the first month, she was annotating roughly 60% of AI outputs with corrections. By month four, that number was 11%.
The AI hadn't improved that dramatically. Her review behavior had changed.
"After a while you start to trust it," she said. "The early stuff had a lot of problems. Now it mostly looks right."
Mostly right is not the same as right. In production budgets, the errors that survive review aren't the obvious ones — they're the ones that look plausible until you're on location and the number doesn't match reality.
The AI had trained the reviewer to trust it. That's a different kind of risk than the AI making errors.
THE PATTERN ACROSS BOTH CASES
In both deployments — and in the Snowflake incident — the same sequence plays out:
AI produces output. Output looks correct. Reviewer approves. Time passes. Error surfaces downstream.
The error isn't in the AI output alone. The error is in the mismatch between what the review step was designed to catch and what the review step actually catches when a human is reviewing AI-generated content at volume.
Standard review processes were designed for human-produced work. Human-produced work has a different error profile than AI-produced work. Humans make errors of knowledge, judgment, and attention. AI makes errors of plausibility — outputs that are internally coherent, well-formatted, and wrong in ways that don't announce themselves.
A 23-second review catches the second kind of error at approximately the same rate as no review at all.
The human in the loop is not a quality gate. It is a latency gate — it slows down how quickly errors reach the client. That's not nothing. But it's not what anyone thought they were building.
WHAT THE REVIEW PROCESS ACTUALLY NEEDS TO BE
The instinct after reading this is to say: require longer reviews, add checklists, increase oversight. That's the wrong fix, because it treats this as an attention problem. It isn't.
First, separate format review from content review — and assign them to different people or different moments. Format review (does this look right, is the structure correct) is fast and AI can help with it. Content review (is the substance accurate, does this match the underlying source data) is slow and requires domain expertise. Collapsing both into one approval step produces a process that does neither well.
Second, build error-detection into the output, not into the reviewer's judgment. In the staging deployment, we added a mandatory source-citation step: every measurement in a staging recommendation had to trace back to a specific room in the property file. The coordinator wasn't reviewing for accuracy — the system was enforcing traceability. Errors that couldn't be traced were flagged automatically. The coordinator reviewed flagged items.
Third, audit review behavior, not just review presence. It matters that a human approved the output. It matters more whether that approval was substantive. Review time, annotation rate, override frequency — these metrics tell you whether your human-in-the-loop is actually functioning as one. If your reviewer is processing 40 outputs in a shift and catching nothing, the review step is providing compliance theater, not quality assurance.
ONE THING WE MIGHT BE WRONG ABOUT
The frame above assumes that the solution to shallow AI review is better-designed human review. But there's a version of this problem where that's not the right answer at all.
In some workflows, the volume is too high and the domain expertise too scarce for meaningful human review at every step. In those cases, the honest answer might be: move the human upstream, into defining the rules the AI operates by, rather than downstream, into reviewing every AI output.
We haven't fully worked out when this tradeoff is appropriate. The risk is that upstream rule-setting gives you the appearance of control without the substance — the rules look comprehensive, the AI follows them, and the edge cases that the rules didn't anticipate still slip through, with no human review step left to catch them.
What we're confident about: a nominal human review is worse than no human review. If the step isn't actually catching errors, it's creating false confidence that someone is watching. False confidence is more dangerous than acknowledged uncertainty.
Working notes from B2B AI deployment in North America. Part of an ongoing series on what we keep noticing across wildly different industries — and what the industry isn't ready to say out loud. More at [zenaicorp.com](https://zenaicorp.com/en).
