I built an AI prediction tool that refuses to guess, 63% win rate, publicly including the losses
So I built this thing called Call It. You ask it a question about the future, will the Fed cut rates, who wins the election, whatever, and it gives you a probability plus every source behind it. You can slide the weights around if you trust the market more than the model, or add your own source.
But that's not really why I'm posting.
A few weeks ago someone asked my app which team LeBron would sign with. It said 50% and gave three reasons. Home court advantage. Young core needs a veteran. Playoff seeding. Looked completely legit.
None of it was real. The question was about a contract. There was no game. No home court. And when I refreshed the page it made up different reasons. Same confidence. Brand new nonsense.
That freaked me out a little. My whole pitch is "no black box" and my app was quietly making things up.
So I changed how it works. The number now gets calculated from actual market data before the AI is even called. The AI just explains a number it can't touch. And if there's no real data, it writes nothing. It just says so and suggests something it can actually answer.
It says "I don't know" a lot more than most AI apps. I've made my peace with that.
Where it stands: 63% win rate on 46 resolved predictions. The record is public and I left the losses in. There are two on there that literally say "we called 87% and were wrong." A daily job resolves predictions automatically now, so I can't conveniently forget the bad ones.
Built solo, nights and weekends around a full time job. I've been writing about it as I go, mostly the parts I got wrong.
Would genuinely like to know what breaks when other people use it.
App: https://trycallit.vercel.app
The writeup: https://medium.com/@anujagadde18/my-app-was-looking-straight-at-the-answer-and-seeing-nothing-4ac9a553a7b9
The structural change here is the part that interests me most.
You’ve moved from asking the model not to fabricate to designing the system so it simply doesn’t have the authority to invent the underlying number.
But as Call It gets more real-world usage, how are you thinking about verifying the system one level above that?
For example, periodically sampling actual predictions, source evidence and explanations to independently check whether the system behaved according to the rules you intended — rather than relying on the same system to demonstrate that it behaved correctly.
That independent production-evidence layer is what I’m building with OpsWatch.
Do you think that kind of external verification becomes useful once users start making higher-consequence decisions from the predictions?
Fair question, and I've been thinking about a narrower version of it.
Right now my verification is structural rather than observational. The number is computed before the model is invoked, so the model has no path to the probability, and there's a gate that returns early when there's no real data. The failure modes I can rule out are ruled out by construction, not by checking after the fact. What that doesn't cover is the interesting part. The model still writes the explanation, and nothing currently verifies that the explanation matches the number it's explaining. It could narrate a 63% as though it were a near certainty and I'd have no automated way to catch it. Same for sources. I show which sources fed a number, but I don't independently confirm the reasoning actually drew on those rather than on the model's own priors.
So yes, sampling real production output and checking it against the intended rules is the layer above what I have. Whether I need it yet is a different question. I'm at 46 resolved predictions and nobody is making consequential decisions on this. Building external verification before I have usage would be solving a problem I can't observe.
What would change that is exactly what you said: consequence. If people start sizing real money on these, the gap between "the number is trustworthy by construction" and "the explanation around it is trustworthy" stops being academic.
Honest answer on where it sits for me though. I ran my Brier score while pulling an export yesterday and it's 0.2527, which is essentially coin flip territory. So my more urgent problem is that the numbers themselves aren't carrying much information yet. Verifying the explanations around numbers that aren't good enough would be premature. Curious what OpsWatch actually samples. The output only, or the inputs the model received too? The second one seems much harder and much more useful.
The second one — where the evidence is available, I’d want to sample both.
Looking only at the final output can tell you that something went wrong, but not necessarily why.
For a system like Call It, I’d ideally preserve enough evidence to reconstruct the relevant decision path for a sample: the input/source material available to the system, the computed probability or structured result, the context passed to the model, the resulting explanation, and the final user-visible output.
Then the assurance question becomes more specific.
Did the explanation accurately represent the 63%, or did the language overstate confidence?
Did the cited sources actually support the underlying evidence/context the model received?
Did the system introduce claims that weren’t supported by those inputs?
Did it stay within the behavioural rules that were agreed upfront?
And where something fails, can we distinguish between a bad source/input, a transformation problem, or the model narrating the evidence incorrectly?
I wouldn’t necessarily collect every interaction forever. The model I’m working toward is risk-based sampling: enough production evidence to independently test whether the system is continuing to behave within its intended boundaries, with heavier sampling around unusual, high-impact or failure-prone cases.
I also agree with you on timing. At 46 predictions and without consequential usage, I wouldn’t argue that you need an assurance layer today.
But if people eventually start putting real money behind the predictions, I think the evidence trail you’re already building would make Call It a particularly interesting system to assure.
This is a useful decomposition, and the failure taxonomy at the end is the part I hadn't structured properly. Distinguishing a bad source from a bad transformation from the model narrating correct evidence incorrectly matters, because those have completely different fixes.
On what I currently retain: the computed probability, the components that produced it, and the source list are all stored. What I don't keep is the exact context passed to the model at generation time, so if an explanation overstated a 63%, I could see that it did but not reconstruct why. That's a real gap and a fairly cheap one to close.
The specific check I'd want first is your second one, whether the explanation's language matches the number. My architecture guarantees the model can't change the probability, but nothing stops it from narrating 63% as though it were near certain. That's the failure mode my design doesn't cover, and I only noticed it clearly because of this thread.
Risk-based sampling makes sense too. Heavier sampling around high-confidence calls specifically, since that's where my Brier score says my errors are concentrated.
Appreciate you being straight about the timing rather than pitching it as urgent.
Thanks — that’s exactly the kind of bounded failure mode OpsWatch is designed to test independently.
For a first check, I’d keep it deliberately narrow:
That would give you a clean answer to one specific question: when the system says 63%, does the explanation actually sound like 63%, or does it read like certainty?
It also sounds like retaining the exact generation-time context would close the biggest evidence gap for this kind of assurance, and as you said, that appears relatively cheap to add.
If useful, I can sketch what a very small pilot would look like around just this one control.
Jason
Two of those four are done, which I did not expect when we started this thread.
The narrative-strength check is in. Generated explanations now get compared against the computed probability, and language that overstates it gets flagged and the verdict line rewritten. Certainty words on anything under 90% trip it, strong-favourite words on anything near a coin flip trip it, and legitimate confident language at 96% passes untouched. So the specific question you posed, does a 63% actually read like 63%, now has a mechanical answer at generation time.
I also shipped confidence intervals on the calibration curve after the other thread pushed on it. Each band shows the range the true rate could plausibly sit in given how thin the sample is. At five results that range is roughly 58 points wide, which is the honest picture rather than a clean line.
The gap you keep pointing at is real though, and it is the one thing I have not closed. I do not retain the exact context passed to the model at generation time. So my check can tell me the wording overstated the number, but not whether that came from a bad source, the transformation, or the model narrating good evidence badly. Your third distinction is the one my logs cannot currently answer.
On the pilot: I would rather not take that on yet, and it is a timing thing rather than a value thing. Nobody is putting money behind these and my Brier score is still at 0.253, so my real problem is that the numbers themselves are not yet beating a coin flip. Assuring the narration around numbers that thin feels premature.
What I will do is add generation-time context retention, since you have now made the case for it twice and it is cheap. That way the evidence trail exists whenever the question does become live.
Thanks for the detail in these, genuinely more useful than most product feedback.
That makes sense, and I agree that the underlying prediction quality has to become decision-useful before assurance around the narration becomes fully valuable.
I would push back on one point though: the commercial side is becoming real. People are already looking at paid partnerships and paid bounded evaluations around independent AI assurance, so this is no longer purely a hypothetical future category.
That is why the context-retention change matters now. Once you retain the exact evidence/context supplied at generation time, you can distinguish:
bad source evidence → transformation error → model narration error
rather than only knowing that the final wording overstated the probability.
You have already closed two of the four issues, and this third one gives you the evidence trail needed for a much stronger assurance boundary later.
I agree that a pilot on your product specifically may still be premature while the Brier score is at 0.253. But I would separate that from whether there is a real paid market for this kind of independent assurance — there is.
So I would leave the pilot itself for now, keep building the evidence trail, and revisit it once the prediction layer improves enough that users are making consequential decisions from the outputs.
At that point, the assurance problem becomes immediate rather than theoretical.
Built it. Every analysis now stores the source context, the computed probability and which path produced it, the context handed to the model, the explanation returned, and the consistency verdict. So the three-way distinction you described is answerable now rather than in principle.
Two practical choices in there. Traces prune themselves, clean ones after two weeks and flagged ones after ninety days, since the flagged ones are the only ones worth opening. And reading them needs an admin key, because a trace is essentially the whole generation pipeline for one question.
Fair point on the commercial side, and I'll take your word for it since you're closer to that market than I am. My read stands for my own situation rather than the category: at 0.253 my problem is upstream of narration quality.
Agreed on where we've landed. The trail exists now, so if the prediction layer gets good enough that people act on it, the evidence is already there rather than needing to be retrofitted.
That closes the attribution gap nicely.
You now have enough retained state to reconstruct the important chain:
source context → probability/path → model context → explanation → consistency verdict
So if something fails later, the question is no longer simply “was the explanation wrong?” You can actually locate where the divergence entered the pipeline.
The retention split also makes sense. Keeping flagged traces longer gives you the evidence that is most useful for investigating failures without retaining every clean execution indefinitely.
The next boundary I’d keep in mind — not necessarily something you need to build now — is evidence integrity. Once these traces become important enough to support a consequential decision or an external assurance result, I’d want to establish that the retained generation-time evidence could not have been silently altered after the event, and that pruning itself leaves an auditable record.
But I agree with your sequencing. At 0.253, improving the underlying prediction layer is the immediate problem. You’ve now done the useful thing on the assurance side: the evidence trail will already exist when the outputs become consequential enough to warrant independent verification.
I’d leave it there and focus upstream.
Agreed on evidence integrity being the next boundary, and noted for whenever traces start supporting something consequential. Hash-chaining writes and logging pruning as events rather than silent deletes is the obvious shape, and it's much easier to add before there's history worth protecting than after.
Taking your closing advice. The prediction layer is the problem, and I know roughly where: my errors concentrate in the 80 to 90 band, so the model is producing numbers that are too extreme rather than being wrong about direction. Several of those 85% calls should have been 65% calls.
Thanks for the thread. Three concrete changes came out of it and I'd have found none of them on my own.
That's a good place to leave it.
The 80–90% concentration is useful because it gives you a much narrower problem to attack than general prediction accuracy — calibration of confidence rather than direction.
And yes, I'd add the trace-integrity pieces before you need them. They're cheap architecture now and potentially expensive evidence reconstruction later.
Glad the thread produced something concrete. If the prediction layer reaches the point where people start relying on these outputs for consequential decisions, that's when I'd revisit the independent assurance side. By then you'll already have most of the evidence trail needed to test it properly.
Good luck tightening that 80–90 band.
The architecture fix is deeper than you may be crediting: the AI is never the source of the fact, only the narrator of it. That's what makes any AI-over-data product trustworthy. You found the general rule, not just a patch.
But the harder question a sharp user will test: win rate is almost the wrong headline metric for a "refuses to guess" forecaster. 63% on 46 tells me little without base rates. 63% on coin-flips is barely above chance; 63% calling confident longshots is excellent. Same number, wildly different products.
What proves your honesty claim isn't win rate, it's calibration: when you say 87%, does it resolve yes 87% of the time? A well-calibrated 55% forecaster beats a lucky 63% one, you can size decisions on it. Your two public "called 87% and were wrong" losses are exactly the data that shows it, plot predicted probability against actual resolution rate.
Tracking calibration yet, or just win/loss? The calibration curve is the moat, not the hit rate.
This is the most useful comment I've gotten. You're right, and it's a gap.
Right now it's just win/loss. Which as you say tells you almost nothing on its own. I've been quietly aware the headline number is doing more work than it deserves.
The good news is the data is there. Every prediction stores the confidence we gave it and how it resolved, so bucketing by confidence band and plotting predicted against actual is genuinely buildable. That's the next thing I'm doing.
One honest caveat: with 46 resolved I don't have enough in each bucket for the curve to mean much yet. Two or three predictions in the 80 to 90 band tells you nothing. So I'd rather ship a thin curve that's clearly labeled as thin than a smooth one that implies more confidence than the sample supports. It'll fill in as things resolve, and there's a daily job settling predictions now so it should build on its own.
And yeah, the two "we called 87% and were wrong" entries are the most useful things on that page. Those are the ones that tell you whether the number means anything.
Appreciate you pushing on this. Will come back when the curve is up.
Your small-sample instinct is exactly right, and it points at a better tool than the bucketed curve for now. A calibration curve needs density per bucket to mean anything, which you don't have at 46. A Brier score does the same job in a single number and stays meaningful at small n, because it scores every prediction individually instead of needing a populated bucket. Accurate and calibrated in one figure, and it degrades gracefully with sample size instead of going noisy. Ship that as the headline honesty metric now, let the curve fill in behind it as a visual once buckets populate.
The deeper move, on-brand for "refuses to guess": report the calibration with its uncertainty, not just a point. A thin curve with visible error bars is more honest than a clean line, and it turns your weakness (small sample) into a feature, we show how sure we are about how sure we are. Most tools hide their confidence intervals. Yours showing them IS the pitch.
The two "called 87% and were wrong" entries matter precisely because a well-calibrated forecaster is SUPPOSED to be wrong 13% of the time at 87%. If they never lost an 87% call, the number would be a lie.
When you plot it, are your high-confidence calls landing above or below the diagonal? That over/under tells you which way to correct.
Shipped the Brier score, and you were right that it was the better call at this sample size. It's the first thing on the accuracy page now.
0.253 on 46 resolved. Saying 50% to everything scores 0.250, so I'm essentially at the line. The 63% win rate came from calling likely favourites where being right was easy, and a handful of confident misses wiped out the advantage.
To your question about which side of the diagonal: above it, meaning overconfident. The damage is concentrated in the 80 to 90 band. An 87% that missed costs about 0.76 on its own, while a 98% that landed earns back 0.0004. Five confident misses in that band are doing almost all of the work.
The correction is obvious in hindsight. Several of those 85% calls should have been 65% calls. Same outcomes, much better score.
Error bars are the next thing. You're right that a thin curve with visible uncertainty is more honest than a clean line, and it fits the product better than hiding the sample size does.
https://trycallit.vercel.app/accuracy
The strongest part is separating the prediction from the explanation. If the AI cannot change the underlying probability and has to stay silent without real evidence, the system becomes much more trustworthy.
Thanks. That separation took me longer to arrive at than it should have.
The part that surprised me was how much stronger it is as a structural rule than as a prompt. Telling a model "don't make things up" is a request it can ignore. Computing the number first and handing the model something it has no power to change means there's nothing left to invent. Same goal, completely different reliability.
The staying silent part was harder to accept than I expected. It felt like the app was doing less. It's doing less confidently wrong things, which turns out to be the version I want.