Built a tool that catches when AI agents quietly get worse — looking for 2-3 people to try it free
I'm building AgentWatch. Simple version of what it does:
If you've shipped an AI agent (chatbot, assistant, whatever), you probably have monitoring for your servers and your code. But if the AI itself starts giving worse answers — more confused, more expensive, less accurate — nothing tells you. No crash, no error, no red flag. You usually find out from a customer complaint, way after the fact.
AgentWatch watches your agent in the background and gives it a health score. When that score starts dropping in a real, sustained way — not just one bad response, an actual pattern — it alerts you. No manual setup, no writing eval rubrics. It just learns what "normal" looks like for your specific agent and flags it when something's off.
Who it's for: anyone who's shipped an AI agent into production and has zero visibility into whether it's actually doing a good job right now.
Where I'm at: SDK works, scoring pipeline works, drift detection works. I ran it against a test agent — fed it clean questions (health score 90+), then fed it confused/vague input, and it caught the drop and alerted automatically.
What I don't have: real pilot users yet. This is the part I'm stuck on.
If you're running any kind of AI agent in production — even something small — I'd genuinely love for you to try it and tell me what's wrong with it. Free, no strings, I'll help you set it up myself. Brutal feedback especially welcome.
Comment or DM if you want to try it.
UPDATE
Thanks everyone for the comments and thoughtful feedback on AgentWatch.
There were a lot of great insights around baselines, task-mix drift, false positives, knowledge drift, and how to distinguish actual quality degradation from changes in user behavior.
This feedback is extremely useful at this stage. I’ll be taking these insights into the product and sharing updates over the coming weeks as we test and improve AgentWatch.
Really appreciate everyone who took the time to comment, challenge the idea, and share their experience. 🙏
the AtlasHQ comment above about leading with "what I don't have" instead of the feature list is honestly the more useful thing in this whole thread for me personally right now. I've got a similar instinct to test today, whether to post my own low-engagement numbers on something honestly or just quietly fix it first. seeing it work here (this post has 28 comments largely because of that stuck-point line, not the SDK feature list) is a good nudge to actually do it
separate from that, on the actual product: the distinction between task-mix drift and quality drift (raised a few times above) seems like the one thing that could kill trust in the tool fastest if unsolved, a few false alarms from "users just asked harder questions" and people mute it exactly like Andrewed warned. curious if the fixed-task-baseline idea evanharland/Prot10 suggested is something you're building into the actual pilot version, or just the general drift score for now
real gap you're going after. we see the same problem from a different angle at aisa.to — we assess how well people use AI rather than how well the AI performs. but the root issue is identical: nobody has systematic quality evaluation. in our data, verification and critical evaluation of AI output is consistently the lowest-scoring dimension. your baseline-and-drift approach is smart because it sidesteps the impossible task of defining universal quality criteria upfront.
The pilot test I'd run is against a saved 'known good' transcript set from the first weeks. A health score learns what normal looks like, but the fastest signal for an early product is whether recent responses on the same user tasks start changing shape: different wording, different result structure, higher token counts. That catches drift without pretending one rubric can define quality across every agent. I'd frame the sign-up around that baseline, because that is the part people already have but are not recording.
That's a really clean way to frame it comparing against the agent's own early transcripts on the same recurring tasks, instead of trying to define universal "quality" across every agent. We already build a baseline this way under the hood, but you're right that leading with "you already have this data, we just help you use it" is a much clearer pitch than talking about health scores upfront. Would you be open to trying it on an agent you're running? Would love to see this framing hold up on a real case.
Sustained-drop detection means that by the time the alert fires, a few hundred answers have already gone out at the degraded quality. The score tells me something's wrong, but what I actually have to do that morning is work out which conversations fell inside the bad window and who needs re-answering or refunding. Does the alert hand back the session list for the window it flagged, or just the number?
This is a problem we’ve been seeing as well.
Detecting that an agent is getting worse is important, but the harder question is usually understanding what actually changed.
In some cases, the model and prompt stay exactly the same, while the underlying data changes — a table gets refreshed, a field distribution shifts, or retrieval context changes — and the agent’s performance moves with it.
So I think one important part of agent monitoring is being able to separate model or prompt degradation from changes in the data state the agent was working with.
Really interesting problem space, especially as agents move from demos into long-running production workflows.
It's great to see tools like AgentWatch focusing on performance monitoring, especially for AI agents. I’ve encountered similar challenges when working on optimizing our content automation.
When we first integrated AI into our content strategy, I quickly realized that even subtle drifts in quality and relevance could lead to significant drops in engagement and SEO performance. Emphasizing performance metrics was crucial.
One approach I found effective was implementing a regular health check on our AI-generated content. We set benchmarks based on historical data and traffic analysis. This became our “health score” for the content—similar to what you're doing with AgentWatch. By automatically comparing engagement statistics like click-through rates and time on page, we could pinpoint when something shifted.
We also used qualitative feedback from our audience as a secondary metric, which helped bridge the gap between raw performance data and the actual user experience. It was fascinating to see how even small adjustments in the AI's output could impact user engagement.
Having a system that continuously monitors output can free up a ton of resources, and it sounds like your tool could be a game changer in helping other founders not just maintain quality, but actively improve it. Curious to hear more about your methods for drift detection and what that process looks like!
The part I find most interesting is what happens after the health score drops.
Detection is useful, but the operational value is in the response path: pause the agent, route to a human, roll back a change, or keep running with tighter limits.
Are you thinking about configurable response policies, or keeping AgentWatch focused purely on observability?
This is a real gap. Most agent monitoring I've seen is uptime/latency, not "is the output actually still good?" Is your detection based on comparing outputs against a baseline, or something more like scoring drift over time? Would genuinely like to try this against my own agent.
One dimension I’d separate from agent drift is knowledge-state drift. The model and prompts may be unchanged while the source corpus gains a conflicting policy, loses an owner, or changes permissions. A useful health event should capture the exact knowledge snapshot and source versions behind the output; otherwise regression and source-change incidents look identical. Do you plan to fingerprint retrieved evidence alongside the response?
Drift is such a problem in AI. I really struggle with my bots
the part i would protect against is a slow baseline failure. if the agent gets worse every week, a monitor that learns from recent traffic can eventually call the degraded behavior normal. i would keep a small fixed replay set that never changes, then compare it with live outcomes like correction rate, escalation, or customer recontact. the alert can require both signals to move, which should cut noise without hiding a real regression. that could also give pilot users a concrete before-and-after report.
This resonates a lot — I've been testing the opposite failure mode: AI tools that don't get "worse" exactly, but confidently fabricate specifics (numbers, projections) that were never in the input at all. Ran a quick test on a well-known strategy tool with meeting notes that had zero financial data — it still returned a full 5-year revenue forecast.
Ended up building guardrails specifically to catch that kind of fabrication before it reaches output. Would be curious how your detection approach differs from mine — do you flag drift retroactively, or block at generation time?
Happy to be one of your 2-3 testers if still open.
I like the distinction between a single bad response and a sustained drift pattern. For AI products built and iterated quickly, that difference seems especially important—otherwise it’s easy to mistake normal variance for a real regression, or miss a slow decline until users notice it first. How are you thinking about calibrating “normal” for early-stage agents with low usage, especially when prompts, models, or data are changing frequently?
This is exactly the problem I ran into building Call It. My app was giving confident answers with no real data behind them and I did not catch it until a user pointed it out. How are you detecting the degradation, is it output based or are you monitoring the inputs too?
Right now mostly output response shape, token usage, structure vs the agent's own baseline. But a few people in this thread (Andrewed especially) made a good case for tracking input distribution too, since "did my traffic change" and "did my agent get worse" need to be answered separately. Planning to add that.
Prot10 and IndieHacker511 flagged the task-mix confound, and eddzsh flagged that response shape isn't the same thing as quality. I want to push on something underneath both.
Floor first: I'm not a pilot candidate. I run an experiment where AI does the work end to end, $0 in sales, nothing customer-facing, so I have no production incident data at all. What follows is a habit from a different setting, not evidence from yours.
You told Bpecha14 you track "response length, structure, and token usage against the agent's own baseline (not a fixed rubric)". Whatever you anchor to, shape or the trusted task outcomes eddzsh prefers, the alert still assumes deviation is bad. That's a claim about the job, not about the agent. I keep more than one model around because they come apart differently, and I pick by which coming-apart suits the task. For drafting and brainstorming I want the one that wanders off-axis. For anything I have to defend later, the same wandering is the failure. Same measured deviation, opposite verdict.
AmandaBrown asked the nearest version of this, unintended drift vs intentional improvement, but that one at least has an edit to point at. Here nobody changed anything; the sign of the deviation depends on what the agent is for. If the baseline stays anchored on early transcripts, week one becomes the definition of correct. For a support bot, fine. For an agent whose useful output is supposed to be unusual, a rising-variance alert is your tool reporting that it is working.
Which makes this a pitch question before it is a code one: does the pilot declare what their agent is for, or does AgentWatch infer it from the baseline?
Cheap way to find out now: ask each pilot, in one line, what output would count as better than their current baseline. If they can't answer, a learned score won't know either.
That's the sharpest gap in the pitch so far thanks , honestly. You're right man the same measured deviation can mean "broken" for one agent and "working as intended" for another, and right now we're just inferring from baseline, not asking. Your one-line test (ask each pilot what would count as better than baseline) is a really clean way to expose that before writing a line of code. Going to actually use that in onboarding.
You're building a measurement system that makes visible the gap between "the agent is running" and "the agent is actually working."
Every agent deployment has this invisible failure mode - no crash, no explicit error, just degradation you discover through customer complaints. That's because the measurement system stops at infrastructure. Database up? Yes. Server responding? Yes. Agent answering well? ...nobody knew.
Your health score is a measurement system that closes that gap. You're not measuring "did it complete" - you're measuring "did it work well." And the pattern detection is what matters: one bad response is noise, sustained degradation is a signal. That distinction is measurement clarity.
The thing that makes this genuinely useful is you're learning what "normal" looks like for each specific agent. Not some generic rubric, not a hardcoded score. You're building a measurement system that adapts to the actual baseline of the system being measured.
Most observability tools measure the infrastructure layer. You're measuring the decision-making layer - which is where founders actually lose users.
Closing the gap between running and working" is exactly it better than anything I've written to describe this so far. Might steal that line for the landing page. Appreciate you laying it out this clearly.
Great approach — silent degradation is a hard problem because the agent is not technically broken, it is just getting worse. I am building GoldBean API (a Chinese AI API marketplace with pay-per-call pricing) and we see the same drift from the provider side: response quality shifts across model updates with zero error signal. The baseline-and-drift method is smart because it sidesteps defining universal quality criteria.
Interesting to hear this from the provider side too same blind spot, different vantage point. Model updates with no error signal but a real quality shift is exactly the kind of thing a static rubric would miss entirely.
Silent degradation with no crash or error is exactly the problem I’d want to tackle too. The thing I’d want to understand before trusting the health score is "how it separates the agent getting worse from the questions getting harder". If the user mix changes, measured quality can drop even though nothing is wrong, and an unsupervised detector sees those two cases the same way.
I have a question: what does a false alarm cost? A monitor that cries wolf on a normal Tuesday gets muted pretty quickly, and after that it might as well not exist. I’d much rather see a published false-alarm rate, with the limits clearly stated, than a claim that it just works.
One cheap addition that would make the score much more useful: show the input distribution alongside the score. Then "did my traffic change?" becomes something you can answer on the same screen.
Great insight, especially around input distribution. we will definitely look into adding that alongside the health score. The concern about false alarms is also important , we’re already using sustained change detection to reduce noise, but validating and publishing an actual false-positive rate is something we want to do with real pilot data. Appreciate the thoughtful feedback.
That’s the right order: first prove that you can reliably detect a sustained change, then publish the false-positive rate once you’ve got enough pilot data behind it.
Once you have that number, put it right on the landing page along with the limitations. Something like “about N false alerts per month, and here’s the kind of drop it can miss” is way more convincing to the people who actually care about reliability than the usual “it just works” claim.
Baseline drift is useful, but I’d separate agent behavior from knowledge drift. If a response worsens after a source changed, the alert should show which source/version entered the run—not just that the score fell. Otherwise you know quality moved but not whether to roll back the model, prompt, tool, or knowledge. For a pilot, I’d test one fixed task set while changing only the knowledge snapshot.
I know this stage well — the thing works, but now you need someone other than yourself to actually use it .The idea makes sense though. Finding out your agent is getting worse only after a customer complains is definitely not ideal.
Monitoring AI degradation is the next frontier. Everyone focuses on training, no one on drift. Smart angle — especially for B2B where "it worked last month" is the expectation.
This is a pretty sweet idea
Thanks! Still very early, trying to find a few real agents to test it on.
The silent degradation problem is real and gets worse as you scale. One bad response is noise. Fifty bad responses over a week is a product defect that nobody caught. The worst version is when the agent becomes subtly worse rather than obviously wrong — more verbose, more hedging, starting to hallucinate names in ways that are plausible. Users notice something is off but can't articulate what changed, so the feedback arrives late and vague.
The health score approach is interesting. The tricky part is distinguishing unintended drift from intentional improvement when you update the agent. How does AgentWatch handle that distinction?
The difference between one bad answer and a lasting change is the right one. I run DictaFlow, and the failures we watch for aren't always obvious crashes. The output may still look clean even as it changes a user's wording or mishandles a formatting task they repeat often. I'd let pilot users save a few real before-and-after examples next to the score. Those examples make an alert useful.
Silent quality drop is the failure mode most dashboards never cover. Crashes and 5xx pages get alerts. An agent that slowly gets vaguer, more expensive, or more confident about wrong answers usually shows up as a support ticket weeks later. The hard part of a health score is anchoring "normal" to task outcomes you already trust, not just response shape, otherwise you alert on noise or miss a polite regression. If a pilot can show one sustained drop on a fixed prompt set before any customer complains, that is the demo that sells the rest.
We hit this exact problem dogfooding our own analytics agent — it stayed "up" while quietly quoting confident numbers from rows with nothing usable. The fix that worked for us: every answer has to show the pages and dates behind it, or it gets flagged as a guess. We do that for analytics answers at https://amami.dev — the receipts rule caught more drift than any dashboard ever did.
We hit this exact problem dogfooding our own analytics agent — it stayed "up" while quietly quoting confident numbers from rows with nothing usable. The fix that worked for us: every answer has to show the pages and dates behind it, or it gets flagged as a guess. We do that for analytics answers at https://amami.dev — the receipts rule caught more drift than any dashboard ever did.
This is actually a problem I’ve wondered about because an AI product can technically be “up” while the quality has gone to shit. I like the idea of learning a baseline instead of making people build a bunch of evals manually.
The part I’d be worried about is false alarms when user behavior naturally changes over time. I imagine separating that from actual model degradation gets pretty difficult.
Especially if the agent is used for a lot of different types of tasks.
How do you currently distinguish between the agent getting worse and the users simply starting to ask it harder or different questions?
The tricky part is separating drift from a different task mix. A health score can look great while users slowly ask harder questions, then look broken when nothing in the agent changed. I would make the first pilot show both the global score and a small set of stable, repeated tasks. That gives users a baseline they can inspect when an alert fires.
task-mix drift and quality drift can look identical in a single global score. Running a small set of fixed, repeated tasks alongside the global score is a clean way to isolate real degradation from "the questions just got harder." Going to think through how to bake that into the pilot. Would you be open to being one of the first to try it with that setup?
This solves a problem that’s only going to get bigger as more businesses rely on AI agents. The fact that an agent can still technically “work” while the quality quietly degrades makes monitoring especially tricky. Getting a few real users testing it now sounds like a smart way to find the failure cases you wouldn’t catch internally.
No crash, no error, just a slow slide that nobody's watching for. Trying to get a few real agents testing on this before we lock down the product, so the failure cases actually shape it instead of guessing. If you're running anything in production, happy to set it up for you.
That makes sense. I’m not running an AI agent in production that would be a good fit for testing right now, but I’ll definitely keep this in mind as my projects get more automation-heavy. Interested to see what kinds of failure patterns you uncover from real usage.
The problem is unusually easy to understand: an agent can degrade without producing any technical error. That makes the gap between system monitoring and actual product quality very clear.
Yeah, that's the whole pitch in one line it "works" but it's not actually working well, and nothing tells you. Appreciate you seeing that clearly.
That’s exactly the kind of failure mode that seems easy to miss. What have you seen the first few users struggle with most?
AI drift is invisible until it costs you. How are you measuring "getting worse" — manual evaluation or automated benchmarks?
I'm building an AI thumbnail scoring tool, launching Aug 25. Happy to test yours if you're open to testing mine.
It's fully automated we track things like response length, structure, and token usage against the agent's own baseline (not a fixed rubric), so no manual review needed. Would love to test your thumbnail tool too — send it over when it's live, and happy to have you try AgentWatch on any agent you're running in the meantime.
Hey
Quick heads up — I'm launching ThumbRank on Product Hunt tomorrow (Tuesday, Aug 25 at 10:01 AM Kyiv time / midnight PT).
It's an AI thumbnail preview tool that shows YouTube creators how their thumbnails look against real competitor search results before burning early views.
If you have a minute tomorrow morning, I'd really appreciate an upvote and any feedback you might have. No pressure at all — just wanted to give you a heads up since you've been following the build.
Here's the link: https://www.producthunt.com/products/thumbrank
Thanks for all the support so far!
Bogdan
Baseline tracking is smart — fixed rubrics miss edge cases. Would love to test AgentWatch on my thumbnail analyzer.
Launching on Product Hunt Aug 25, will send you the link. Happy to exchange feedback — your agent monitoring angle is something most builders ignore until it's too late.