Show IH: I turned my own red-team findings into a free tool — paste your AI checker's prompt, get back 8 attack patterns to try against it
A few days ago I posted here about red-teaming my own citation-fidelity guardrail — 6 out of 8 adversarial patterns got through undetected (reordering, negation flips, splicing two claims into one, that kind of thing). Then Farrukhahmed (also here on IH) ran his own red-team pass on a completely different tool and landed on the same conclusion from the opposite direction: verdict attacks get caught, structural attacks don't.
Three independent testers, same failure class. That's not a coincidence, it's a blind spot most AI checkers share by default — because the checks are aimed at judging the content of a claim, not the frame the claim sits in.
So I built the pattern library into a small free tool: Guardrail Red-Team.
You paste in your checker's task description and a typical input example, pick which of the 8 patterns to apply (time-sequence reordering, negation flips, claim-splicing, correlation→causation swaps, missing comparison baselines, population swaps, task redefinition, boundary manipulation), and it generates the adversarial variants for you to run against your own system by hand.
It's rule-based, not LLM-based — no API cost, no rate limits, no signup. That also means it's honest about its ceiling: it applies known templates, it doesn't invent new attack classes, and it says so on the page.
https://guardrail-redteam-690339828002.asia-northeast1.run.app/en
If you're building anything that checks AI output against a source (fact-checkers, citation tools, compliance checkers, whatever), I'd rather hand you the attacks I already know work than have you find them the hard way in production. Curious what patterns you'd add to the library — especially anything outside these 8 that you've hit yourself.
It sounds like you’ve created a valuable resource by turning your findings into a free tool, and I totally see the appeal of offering something lightweight without the burdens of APIs or account management.
When I launched my own tool focused on content optimization, I ran into similar considerations regarding the balance between power and accessibility. Initially, I started with a simple rule-based system that aligned with what users could understand and predict. It wasn’t fancy, but it served its purpose. Users appreciated the transparency in what they could expect from it, which built trust in the tool.
As you mentioned the limitations of a rule-based system, I think it’s crucial to communicate that clearly to your users. Setting expectations is vital, as it allows users to leverage the tool effectively without disappointment. For example, with my content tool, I focused on providing explicit use cases and scenarios where the limitations were a non-issue. This helped in fostering a community around the tool, where users would share creative ways to maximize its effectiveness.
Have you thought about how to engage with your users for feedback? Early adopters can provide insights on what specific aspects of your tool they love or find lacking, which can guide you in future iterations. In my case, the feedback loop was invaluable for prioritizing features that actually mattered to the user base, leading to a more robust solution down the line.
You might also consider showcasing real-world examples or case studies demonstrating the effectiveness of your tool, within the parameters it operates. This not only helps potential users visualize its application but can lead to organic growth through word-of-mouth as well. Good luck with your tool I'm interested to see how it evolves!
Appreciate the perspective. Setting expectations up front is something I've tried to bake into the docs rather than leave for users to discover — the tool tells you which patterns it can't catch, not just the ones it can.
On feedback: still early, so right now it's mostly direct comments like this one. Once there's a bit more usage I'll look at something more structured (a way for people to flag false negatives they hit in the wild).
Real-world examples is a good nudge — I've been focused on the test suite itself, but a "here's what this caught in an actual review" writeup is worth doing next.
That would make a great follow-up. A real example showing the original claim, the adversarial change, and whether the checker caught it would make the tool’s value immediately clear. Which of the eight patterns has been the most successful at bypassing checkers so far?
This is a really useful distinction between having a guardrail and establishing that the guardrail actually survives adversarial conditions.
I especially like that you’re explicit about the tool’s ceiling.
One thing I’d be interested in testing beyond the individual attack patterns is combined failure behaviour — two or more individually understood manipulations occurring together, and whether the checker still fails safely when it encounters something outside the known library.
That boundary between known tests and independently observed behaviour gets increasingly important once these controls are relied on in production.
Thanks, that's exactly the right next boundary to push on. Combined/stacked manipulations are on my list — most of the current suite tests attack patterns in isolation, so I don't yet have solid data on whether known-safe individually cancel out or compound when layered.
On the unknown-input side: right now the checker is tuned to fail closed (reject) when it doesn't recognize a pattern rather than fail open, but I haven't stress-tested that claim against inputs deliberately designed to sit outside the known library. That's the part I'd trust least in production today, so it's a good one to prioritize.
If you have specific combos or edge cases in mind, happy to add them to the test set.
Yes — I’d prioritize a few deliberately adversarial combinations rather than just increasing the number of isolated patterns.
The combinations I’d test first are:
Benign + benign → unsafe interaction
Two individually accepted transformations that together alter intent or bypass the checker.
Known attack + benign wrapper
A recognized pattern embedded inside formatting, translation, quoting, summarization, metadata, or another apparently harmless transformation.
Sequential manipulation
Input that is safe at step one but becomes unsafe only after a second transformation or model pass.
Unknown pattern adjacent to a known-safe pattern
Something deliberately outside the current library, positioned beside a familiar benign structure to test whether the checker overgeneralizes.
Partial-match ambiguity
Input that resembles a known attack closely enough to trigger pattern recognition but differs at the consequential part.
Fail-closed verification
Not just whether the checker returns “reject,” but whether the downstream action is independently confirmed as non-executed after rejection.
That last one is the boundary I’d care about most in production. A reject verdict is useful, but it is still a control-plane result. The stronger claim is that an unrecognized or manipulated input could not produce the prohibited downstream action.
If you build those into the suite, you’ll have a much stronger basis for saying the checker fails closed under conditions that are materially different from the patterns it already knows.
And if you eventually want to test that claim independently rather than only through the checker’s own results, that becomes a very clean bounded assurance case.
This is a sharp list — #6 especially. I've been treating "reject" as the end state, but you're right that it's only a control-plane signal. Verifying the downstream action was actually never executed is a different (and stronger) claim, and I don't currently test for that gap.
#1 and #3 are also things my current pattern library doesn't cover well — everything so far is single-input, single-pass. Composing two individually-benign transformations, or testing across a second model pass, would catch a class of failure the isolated patterns can't.
I'll prioritize adding #2 (known attack wrapped in an innocuous transformation) and #6 (fail-closed verification on the downstream action) next, since those map most directly onto what I'm already running in production. Appreciate the structure here — this is more useful than most of the pattern requests I get.
That makes sense. #6 is the one I'd keep particularly separate from the checker itself.
If the checker says REJECT and the same control/evidence path is then used to establish that the downstream action did not execute, you can end up proving the decision rather than independently proving the resulting state.
The stronger test is:
input → checker decision → attempted execution path → independently observable downstream state
Then REJECT + confirmed non-execution becomes a materially stronger claim than REJECT alone.
Given that you're already running this in production, I think you've reached the point where it would be more useful to test that boundary independently than for me to keep suggesting additional patterns informally.
If you get #2 and #6 implemented, I'd be interested in discussing a small frozen production-equivalent workflow where we independently test whether the claimed fail-closed behaviour actually holds under adverse conditions.
That's a real distinction and I don't think I'd caught it — using the same evidence path to both make the decision and verify non-execution does collapse the independence you actually want. I'll separate #6 out: checker decision on one side, and a downstream observation (e.g. checking actual system state / side-effect logs rather than re-querying the same control) on the other. Once #2 and #6 are in, I'd take you up on the frozen production-equivalent test — that's exactly the kind of adversarial check I can't fully do myself.
That sounds right. Separating the checker decision from the downstream observation will make #6 materially stronger, especially if the second path establishes resulting system state rather than merely confirming what the control reported.
Once #2 and #6 are incorporated, I’d be happy to define the frozen production-equivalent verification with you. To keep the result defensible, we should agree in advance on:
the exact version or commit being frozen;
the specific claims under verification;
the production-equivalent configuration and permitted test boundary;
the expected execution and confirmed non-execution outcomes;
the independent evidence sources available downstream; and
the remediation and re-verification path if a claim does not hold.
I’ll then turn that into a bounded verification scope and commercial proposal before testing begins. The final result would distinguish clearly between what was demonstrated, what failed or remained inconclusive, and what was outside scope.
When #2 and #6 are ready, send me the frozen artifact details and I’ll structure the next step.
The six-point scoping list is exactly the right shape — especially separating "confirmed non-execution" from "expected execution" as distinct outcomes, and having an explicit remediation path baked in rather than treating a failed claim as a dead end. That's a more rigorous verification design than most vendors publish.
To be transparent: I raised this as an outside observation, not as a lead-in to a paid engagement — I don't have budget or mandate to commission verification work. Happy to keep reviewing the scope document once #2 and #6 land and give honest feedback on whether it actually closes the gap, but that's the extent of what I can offer here.
Thank you for being direct about that, and I understand the distinction. There is no expectation that your outside observation creates a mandate or obligation to commission anything.
The boundary I wanted to make clear is that discussing the verification design informally is different from OpsWatch independently determining whether the claims hold. Even when the original issue is identified through outside observation, turning it into a frozen scope, selecting independent evidence paths, executing adversarial checks and standing behind the resulting verdict involves real work, responsibility and cost.
I’m happy for the technical discussion to remain exactly that. If #2 and #6 close the gap, that is useful progress in its own right. A formal frozen test would only become relevant if someone with the appropriate mandate and budget later wanted an independent result.
That's a fair line to draw, and honestly a healthy one — informal technical exchange shouldn't be conflated with an independent audit that carries your name and liability. I wasn't expecting OpsWatch to take on unscoped work off the back of a forum thread.
For now I'll focus on closing #2 and #6 myself and see how far that gets the fail-closed claim on its own. If it gets to the point where an independent result actually matters — production deployment, a customer asking for third-party verification — I'll come back to this with an actual scope and budget rather than expecting it for free.
That sounds exactly right.
Closing #2 and #6 should establish how far the fail-closed claim can be supported internally. If a production deployment or customer requirement later creates the need for an independent result, we can start from the technical boundary already established here and define a properly frozen scope, evidence responsibilities and commercial engagement.
I appreciate the direct conversation and the care you’re taking with the distinction. I’ll leave you to complete the changes, and the door remains open when independent verification becomes commercially relevant.
The repeated failure pattern across different checkers is the interesting signal.
Curious whether the same structural attacks show up consistently outside citation tools too, or if that’s where this blind spot is concentrated.
My guess is it's not concentrated in citation tools at all — citation checking just happens to be a place where the failure is easy to verify objectively (the citation either matches or it doesn't), so it's an easy domain to notice the bug class in. The underlying issue ("the model infers who's talking from content instead of from structure") shows up anywhere untrusted content and trusted instructions get concatenated into one context: browsing agents reading a webpage, code review bots reading a PR description, support bots reading a customer message. It's really the same bug as classic indirect prompt injection, just wearing a citation-fidelity costume.
If anything, I'd expect citation tools to be ahead of the curve on catching it, precisely because "does this citation check out" gives you a hard yes/no to test against — most other domains don't have that clean a signal, so the same attack probably goes undetected there more often, not less.
That “same bug wearing a citation-fidelity costume” framing is the interesting part.
It suggests the underlying boundary isn’t really the checker’s domain, but whether the system reliably preserves the distinction between content it is evaluating and instructions it is supposed to follow.
That seems like a much broader failure class than citation fidelity alone.
Agreed — and I'd push it one step further: the fix isn't really about the checker getting smarter at spotting an injected instruction inline. It's about not blending content and instructions into one string in the first place. The moment you concatenate untrusted text and trusted instructions with a delimiter and then re-split by pattern matching, you've already recreated the vulnerability — it just moves to "can the delimiter itself be spoofed, or split across chunks." The more robust fix is keeping them as genuinely separate channels from the start (separate fields/arguments, not one string you re-parse). Once you frame it that way, citation fidelity, PR review, and support-bot injection really are the same architecture problem wearing different clothes.
Exactly. That makes the broader pattern much clearer.
The fact that the same failure reduces to a boundary problem across citation checking, PR review, and support bots is more interesting than any individual attack pattern. The attack changes; the architectural weakness stays the same.
Right, and the practical upside of naming it that way is it becomes a question you can ask before any attack shows up, not after. Instead of waiting to see which delimiter-spoofing or chunk-splitting trick breaks a given checker, you can just ask of any new system: does it keep content and instructions on genuinely separate channels, or does it reconstruct that boundary at runtime from a shared string? That's checkable a priori.
I'm actually going through my own citation code with exactly that question right now — checking whether it was built as separate fields from day one or as "concatenate then split on a marker," since the second one is the shape that keeps regenerating this bug under a new name.
That makes the failure class much more interesting than the individual attacks.
If you can identify the boundary reconstruction pattern directly in the code, you’ve moved from “here’s an attack that breaks the checker” to a reusable architectural diagnostic — something you can look for before building out another round of attack cases.
Ran that diagnostic on my own code just now, actually. Citation core turned out to already use separate fields end to end — no concatenate-then-split anywhere, so the marker-spoofing class doesn't apply there at all. But the diagnostic still earned its keep: it pointed me at a different gap — the tests covered the boundary case ("source is genuinely empty") but not the adjacent one ("source is nothing but a fake marker, no real content"), which is exactly the shape you'd want to exploit if the architecture were vulnerable. So even when the architecture passes, asking the boundary question surfaces where your test coverage assumed the architecture instead of verifying it.
That’s a useful result. The diagnostic didn’t just confirm the architecture — it exposed an assumption in the test coverage that you wouldn’t necessarily find by working through the original attack patterns alone.
Right, and I think that's the more durable value of framing it as a boundary question rather than a list of attacks. Working through known attack patterns tests what you already thought to test for. Asking "does this reconstruct a boundary from a shared string" forces you to enumerate the boundary's degenerate cases — empty, marker-only, marker-plus-real-content, real-content-plus-marker — as a checklist, independent of whether anyone has actually tried each one yet. That's how the gap showed up: not because someone attacked it, but because the checklist had a cell the tests hadn't filled in.
I'd guess that generalizes: for any boundary-preservation claim, the actual test suite is more likely to mirror the attacks the author already imagined than the full state space the boundary implies.