Thinking about an API-based approach for AI evaluation

I would build the API first, but make the evaluation definition immutable for each run.

Input, output and model version are not enough if someone can change the rubric or reference set after seeing the score. In my own benchmark I hash and timestamp predictions before the real-world outcome exists. I score them later.

I would store the rubric version, test-set hash and creation time with every evaluation. The dashboard can read the same record. Otherwise yesterday's 72 and today's 81 may not mean the same thing.

The API-vs-dashboard debate above misses a distinction I had to learn directing Claude to build Alisio: some outputs shouldn't get a score at all, versioned or not. A rubric-based evaluation record still treats every output as reviewable after the fact. That's fine for a support reply. It's not fine for a number a support agent uses to issue a refund, or in my case, a figure a user sees as their days-to-get-paid. I don't let the model touch those, ever. The check happens before the output exists, as a hard rule the app enforces, not a rubric scored afterward. If your API only evaluates outputs that already reached a user, it's already too late for anything irreversible. Worth splitting your rubric into what can be scored after the fact, and what has to be blocked before it ever gets that far.

API-first is the right default, but a single score isn't the right unit. For DictaFlow, each run should keep the raw speech, intermediate transcript, final text, rubric version, and exact failure reason. That way, teams can replay regressions instead of guessing what went wrong. Teams should also be able to define checks for each stage, such as, "Did a mid-sentence correction survive?" The dashboard can then show the traces and failed examples. That gives engineers something they can debug, not another quality number.

API vs dashboard is the wrong first split. The durable thing is a versioned evaluation record: input, output, model and prompt version, rubric, score, reviewer, and evidence.

Expose that record through the API and render the same record in a dashboard. If the API only returns a score, teams cannot reproduce regressions. If the dashboard owns the state, it becomes another silo.

For agent workflows, I’d want API ingestion plus a replay/trace UI, with customer-defined rubrics instead of one universal “quality” number.

We ship an AI pipeline with real per-request cost (speech in, structured notes out), and the evaluation problem we actually have isn't "is this output correct" in isolation, it's whether the summarization step preserved intent from a messy, self-correcting input (someone says a date, corrects themselves mid-sentence, etc). A generic accuracy or hallucination score doesn't capture that; we ended up needing a rubric specific to our own transformation. So if your API returns a fixed set of eval dimensions, we'd probably still have to build the domain-specific grading logic ourselves on top of it, at which point the API mostly saves us plumbing, not judgment. A thin dashboard where we can define our own per-stage rubric, with the API just running it continuously against production traffic, would be more useful to us than a black-box score. Are you leaning toward fixed eval dimensions that generalize across product types, or letting each customer define custom rubrics?

I'd lean API-first, but only if you're honest with yourself about who your first ten users actually are — teams already shipping AI products want evaluation to be invisible, not another tab to check, so friction at integration time matters way more than feature depth early on. A dashboard is easier to demo and sell in the first conversation, though, so there might be a case for shipping a thin dashboard on top of the API just so people can see the value before they trust it enough to wire it into their pipeline. Have you talked to teams who are currently evaluating manually — what does their actual workflow look like right now, spreadsheets, notebooks, something else?

The API vs dashboard distinction is interesting. I’m curious whether teams are already asking for one of these specifically, or if that preference is still mostly hypothetical.