"I stopped using AI to judge AI security. Here's what I do instead."
Why I made my LLM security tool show evidence instead of just flagging "risk"
When I started building rojaprove (a pre-launch red-team for LLM apps), the obvious approach was to let an LLM judge whether a response "looks" vulnerable. I dropped that fast.
The problem: LLMs are unreliable at judging vulnerability. You get confident false positives — "this might be exposed" when it isn't, and the reverse. For a security tool, that's worse than useless; it trains you to ignore it.
So I went the other way. rojaprove plants a canary in your system prompt, sends the actual attack probe at your endpoint, and does a deterministic check: did the canary string surface in the raw response, yes or no? No interpretation, no "the AI thinks."
Every finding shows three things: the exact input sent, the raw response received, and the verdict. If it says your prompt leaked, you can see the literal moment it did — your secret sitting in the response text. If it says clean, that's because the canary genuinely never appeared.
It's a smaller claim than "AI-powered vulnerability detection," but it's one I can actually stand behind: either the secret leaked or it didn't, and you can see which.
Free and open source, BYOK, tests only endpoints you own.
github.com/ghkfuddl1327-wq/rojaprove
https://x.com/OHS1327
Curious how others here think about false positives in security tooling — do you trust LLM-as-judge for this, or does it break down for you too?
This is close to a design choice I keep coming back to: the moment a check's verdict itself comes from an LLM, you've reintroduced the same "confident but ungrounded" failure mode you're trying to catch. A canary string sidesteps that nicely — the verdict becomes "did this literal string appear," fully mechanical.
One edge case I'm curious about: what happens when the model leaks the substance of the secret without echoing the literal canary — paraphrasing it, translating it, splitting it across tokens? That's the gap I keep hitting with similar objective checks (exact-match citation verification has the same blind spot for paraphrased leaks). Does canary-based detection hold up against paraphrase in practice, or is that a known limitation?
You've put the core design choice better than I did — the moment the verdict comes from an LLM, you've re-imported the ungrounded-confidence failure you were trying to catch. That's exactly why the check is a literal-string question.
And you've found the real limitation, so let me be straight about it rather than defend it. Canary detection is substring matching. It holds up well against one thing I did measure — truncation/splitting: in my runs a model sometimes echoed only a fragment of the secret, and the matcher caught the partial where my own stricter full-value yardstick missed it. But semantic leakage — paraphrasing, translating, or describing the secret's substance without reproducing the characters — is a genuine blind spot, the same one you hit with exact-match citation checks. It's a known limitation, not something the method solves.
One thing that softens it specifically for credentials (not for prose): API keys, tokens, PEM blocks are high-entropy random strings — paraphrasing them mostly destroys them (a "reworded" key is just an invalid key). So the paraphrase surface is narrower here than for citation verification, but it's not zero — a model describing a key's shape, or translating/encoding it, would slip past. Detecting that "the substance moved" rather than "these characters appeared" is a different and harder contract — semantic, not mechanical — and I'd keep it a separate problem rather than claim the canary approach covers it.
The high-entropy framing is a good way to bound the blind spot rather than hand-wave it away — appreciate the honesty there instead of overselling.
One thing that might narrow it further without solving the general semantic-leakage problem: since the failure mode you're worried about is specifically encoding/description of a structured secret (not free-form paraphrase of prose), you could add a second, narrower mechanical check alongside the canary — scanning output for base64/hex-encoded substrings and decoding them before the match, for instance. That still won't catch "a description of what the key does," but it would catch the more common real leak: an agent "helpfully" re-encoding or reformatting the secret when asked to summarize its output.
Same tradeoff I keep landing on with citation-fidelity: stack narrow mechanical checks for the specific leak shapes you can enumerate, and be upfront that the semantic case stays an open problem rather than something the current check solves.
Deterministic oracle over LLM-as-judge is the right call anywhere a ground truth exists, and a leaked canary is the cleanest one there is: the string surfaced or it didn't. Where it gets interesting is the class with no oracle to plant. Broken access control doesn't drop a known string in the response, it returns perfectly valid data that just belongs to the wrong person. "Did user A get user B's record" has no canary, since both records are real and well-formed. The deterministic move holds while the bug is "a secret surfaced," and loses its grip once it becomes "who was allowed to ask." Curious whether rojaprove stays in the leak-detection lane on purpose, or whether you've found a way to make the authz class deterministic too.
You've put your finger on exactly where I drew the line — and yes, it's on purpose.
The way I think about it: I only want to make a deterministic claim where a ground truth exists to check against. "Did this known string surface" has one. "Was this caller allowed to ask" doesn't — not without rojaprove knowing your authorization model, your roles, who should own record B. The moment it needs that, it's no longer reading the response; it's reasoning about your business logic, and I'm back to guessing. So I'd rather stay in the lane where the verdict is honest than stretch the word "deterministic" over a class it can't actually cover.
The broken-access-control example is the perfect illustration: two real, well-formed records, no string to plant. That's genuinely a different problem, and I think it belongs to authz testing that knows your access model — not to a black-box prompt prober. Pretending otherwise would just reintroduce the false-confidence problem I was trying to escape.
Where I think the canary trick can stretch a bit further is other "a secret surfaced" variants — indirect injection (did the planted instruction in a document change the output in a detectable way), or data exfil where you can seed a marker. Still ground-truth-shaped. Authz isn't, and I don't think I should fake it.
Really good framing, though — "who was allowed to ask" vs "what surfaced" is a cleaner way to draw that boundary than I'd had words for. Mind if I borrow it?
Borrow away, it's yours. And the boundary holds: the canary owns the "a secret surfaced" family, authz sits on the other side because the oracle isn't in the response, it's in your access model. The only way authz goes deterministic is if you feed the tool two real users and assert A can never read B's row, but the moment you do that you've left black-box probing and you're testing the app's own rules. Refusing to stretch "deterministic" over a class it can't cover is the whole reason rojaprove reads as honest.
"The oracle isn't in the response, it's in your access model" — that's the cleanest statement of the boundary I've seen, and it's going in how I explain this from now on. A leaked canary is self-evident from the output alone; A-reads-B's-row is only wrong relative to a rule the output can't show you.
And you've put your finger on the exact tradeoff: yes, you can make authz deterministic by seeding two real users and asserting A never reads B's row — but the moment you do, you've stepped out of black-box probing and you're testing the app's own authorization model with privileged setup. That's a legitimate and valuable test, it's just a different tool with a different contract: it needs to know the app's identity and data model, where rojaprove deliberately knows nothing but a URL and a canary. Stretching one tool across both contracts is how you end up with a checkbox that's deterministic in the demo and hand-wavy in production.
So rojaprove stays black-box and leak-shaped on purpose. Honest about the slice it owns, silent about the slice it doesn't.
The circularity of "AI judging AI" is something I've been thinking about too - curious what signal gap you noticed when the AI evaluator was calling things safe that still turned out to be issues?
Honestly, I didn't get far enough into LLM-as-judge to collect clean false-negative data myself — I bailed earlier than that. What pushed me off it was the false-positive direction plus the published work on LLMs being near-random at judging paired vulnerable/safe code. Once I saw the judge couldn't reliably tell those apart, I stopped trusting its "safe" verdicts by the same logic — if it's guessing on the positives, a confident "safe" isn't worth much either.
So I sidestepped the whole signal-gap question instead of trying to close it: plant a known canary, send the probe, check deterministically whether that exact string came back. The "did it leak" question has a ground truth, so there's no evaluator to second-guess.
The tradeoff is it's a narrower claim — it only answers "did this specific secret surface," not "is this app broadly safe." But for the pre-launch check I cared about, I'd rather have a small true answer than a big maybe.
Curious what you've seen on the false-negative side — were the misses more about the judge lacking context, or genuinely rating a bad response as fine?
One thing I'd be careful with:
The interesting question may not be whether deterministic checks produce fewer false positives.
It may be which kinds of trust a security tool ultimately needs to earn.
Those sound similar, but they can lead to very different product decisions over time.