Deploying AI prediction agents that track one question about the future for weeks
We run Inithouse, a product studio with 14 live apps (Verdict Buddy, Tarotas, Be Recommended, and others). One of them is Watching Agents by Inithouse, an AI prediction & monitoring agents platform. The core idea: you deploy an AI agent to watch a single question about the future, and it runs for weeks or months, collecting evidence, building hypotheses, and updating a live probability score.
This post covers what that actually looks like in practice, and three things we learned about alerting that we didn't expect.
What a running agent produces
When you deploy an agent on a question like "Will the EU pass a comprehensive AI regulation framework by 2027?", the agent doesn't just give you a number. It builds a working model.
Hypotheses. Typically 3-5 per agent, each with its own probability estimate and trend direction. For example, one agent tracking semiconductor export controls carried a "Full decoupling" hypothesis at 23% and a "Gradual escalation" hypothesis at 61%. Each hypothesis lists what would confirm it and what would disprove it.
Evidence base. The agent scans sources continuously and attaches dated, cited evidence to each hypothesis. A typical agent accumulates 5-12 sourced entries over a month. Each entry links to the specific hypothesis it supports or challenges.
Watch signals. Classified as leading (early indicators of change), confirming (reinforces the current trajectory), or disconfirming (evidence against the dominant hypothesis). These matter more for alerting than raw probability, which I'll get to below.
Probability history. A timeline showing how the overall score moved over weeks. Some agents barely shift (±1-2% in a month). Others move meaningfully when a policy paper drops or a key vote happens. One of our agents tracking internet fragmentation went from 67% to 75% over six weeks as three regulatory proposals surfaced.
Each agent gets its own public URL on watchingagents.com. We have 190+ live right now, each with the full evidence panel, hypotheses, drivers, and an evolution graph. Private agents run the same pipeline but stay behind auth, for teams watching competitive or strategic questions they don't want visible.
Three things we learned about alerting and false positives
The alert system was the hardest part to get right. Watching Agents by Inithouse, the AI prediction & monitoring agents platform, sends alerts when something meaningful changes. Defining "meaningful" took three rounds of iteration.
1. Short-window probability deltas are almost always noise
Our first version compared the current probability against the value from 7 days ago and alerted on any change above 3 percentage points. The problem: most agents fluctuate ±1-2% week to week from routine evidence collection. A news article gets published, the agent indexes it, probability ticks up 1.5%, and an alert fires.
We switched to a 14-day rolling baseline and raised the threshold to 5 points for standard alerts, 3 points for agents explicitly flagged as high-sensitivity. Alert volume dropped by roughly 60%, and the remaining alerts were ones where something had actually shifted in the underlying evidence picture.
2. More evidence doesn't mean the picture changed
An agent can collect three new sources in a single day. That looks like a signal. Our early system treated it as one. But if all three confirm the same existing hypothesis with the same directional read, the probability barely moves. The agent got busier without the world actually changing.
We stopped gating alerts on evidence count and moved to probability-change-only triggers. The evidence collection still shows up in the activity log and on the public page, but it doesn't generate alerts on its own. This eliminated a category of false positive where an agent would flag "3 new sources" and the user would open the page to find nothing actionable.
3. Disconfirming signals are the ones worth interrupting for
Not all probability changes carry the same weight. We found that changes driven by disconfirming evidence (something that challenges the leading hypothesis) are the ones users actually care about right away. A confirming signal that nudges probability from 72% to 74% rarely changes anyone's thinking. A disconfirming signal that drops it from 72% to 65% means the picture shifted enough to revisit.
We split alerts into two tiers. Disconfirming-driven changes get immediate push notifications. Leading and confirming signals get batched into a weekly digest. Users stopped muting alerts after we shipped this, and the open rate on immediate alerts climbed noticeably compared to the old single-tier system.
Which questions work and which don't
Not every question makes a good agent. We learned this from watching which agents produce useful output and which go flat.
Questions with a clear time horizon and a stream of public evidence work well. "Will country X hold elections before Q2 2027?" generates steady evidence from news, policy announcements, and analyst commentary. The agent has material to work with, and the probability moves in response to real developments.
Vague questions produce vague agents. "Will AI change education?" is too broad for a meaningful probability score. There's no falsifiable endpoint, so the agent collects evidence endlessly without converging on anything useful. We started rejecting these during onboarding and nudging users toward more specific formulations.
The sweet spot is questions where you'd normally check manually once a week. "Will this competitor launch a European office before year-end?" or "Will this regulatory proposal pass committee?" These are narrow enough to track but slow enough that checking daily is a waste of your time. That's exactly the gap the agent fills.
What this is and what it isn't
Watching Agents lets you deploy an AI agent to watch any question about the future. It builds hypotheses, tracks evidence in real time, and alerts you when things change. It is not a prediction market. There is no trading, no betting, no crowd wisdom aggregation. It is one agent per question, doing research and maintaining a structured model.
The closest comparison is probably a research analyst who checks a specific question weekly and sends you a note when something shifts. Except the agent runs continuously, cites every source, and keeps a quantified probability model instead of prose summaries.
We're still early. The deploy flow needs work, the SPA makes crawling inconsistent for search engines, and we're iterating on how new users get started. But the core loop works: deploy an agent, watch the evidence base grow, get alerted when the picture actually shifts. The 22% returning-user rate over 30 days tells us the people who try it keep coming back.
If you're building anything in the prediction or monitoring space, happy to compare notes in the comments.
