LLM Knights

This is painfully accurate. I ran into the exact same issue while building an AI-powered app - temperature randomly breaking, reasoning params varying between models, and zero clear docs on what actually works where.

The worst part is you don't find out until runtime. Your app works perfectly with one model, then you switch to another and suddenly everything throws exceptions.

I ended up building a small internal wrapper that normalizes these inconsistencies across OpenAI/Anthropic/Gemini/etc. It's been such a time-saver that we productized it as ModelBridge - one OpenAI-compatible API that handles the param mapping so you don't have to maintain a giant switch statement for every model.

Not here to sell though - just saying I feel your pain. What's your use case? Are you building an app that needs to support multiple models, or are you mostly stuck on one provider?

Per-token sticker price is the wrong unit of account, and it is why every comparison table quietly misleads. At SocialPost.ai our cheapest model per token was not our cheapest per finished output, because verbosity and retry rates differ so much between models. Since you are already building a testing playground, publishing cost per completed task from your own test runs would make this table something nobody else has.

This hits close to home. I'm building a tool that turns podcast transcripts into visual content using AI. The generation cost per piece is real, and it makes pricing the product much harder than I expected.

Seems right now that routers are big business, they're in demand as a tool and as an acquisition target.

I wonder if there's some obvious tie-in here for you to monetise.

https://www.justwatch.com/ offers availability of films across different providers - could you have a view like that? Or another analogy skyscanner.net

LLM cost is definitely one of the biggest challenges for AI products.

I think the key question is whether the AI experience creates enough value that users are willing to pay more than the inference cost.

the “sites either miss models or can’t filter” frustration is painfully real. Shipping a searchable table of 300+ LLMs with price, context, and modality filters is the version of “I’ll just build it myself” that actually helps people.

Provider compare coming is the right next layer — model-level is how you shop; provider-level is how you commit.

What’s the first filter or column people ask for that you still don’t have — latency, rate limits, or something messier like “same model across APIs”?

great, think it's helpful

This is actually a problem a lot of AI builders run into.

LLM pricing changes so quickly that comparing models becomes almost a product decision by itself.

The filtering aspect is probably the most valuable part — founders usually don’t just need the cheapest model, they need the best tradeoff between cost, latency, quality, and use case.

Would be interesting to see filters around things like:

• best models for agents

• coding

• RAG/search

• long context

• structured outputs

Great resource. Keeping this updated is probably the hardest part.

This is a real gap, glad someone built it. We run multi-provider AI features and the pricing comparison part was way more annoying than it should've been, every provider formats it differently and cached/prompt-caching pricing especially gets left out of most comparison pages, like the other commenter mentioned.

Would be genuinely useful if "LLM Knights" eventually tracked cache pricing too, since that's often the difference between a model being cheap or expensive in practice, not just the sticker price per token.

The column I would want is cached input price. Anything with a stable system prompt pays the cache-read rate on most calls, and that can be a fraction of base input, so two models sitting next to each other in your table can be far apart on a real bill. Do the free sources you are pulling from expose it?

the only person worth competing is the person you were yesterday

llm pricing comparison is one of those things you assume exists properly until you actually need it and every page is missing half the models. building it because you kept hitting the wall yourself is usually how the useful ones start.

The feature that would make this a daily tool for me is saved test suites I can re-run when a provider silently updates a model. We run LLMs in production at SocialPost.ai and the pain isn't finding the best parameters once, it's catching quality drift after we've shipped. Logging params per run is table stakes; re-runnable suites with diffs against the last run is the killer feature.

Every time I assign a task, it’s never fully completed. Something always gets skipped or ignored, even when I provide a detailed spec. I have to keep adding things later, which means more costs. It’s frustrating—it honestly feels like it's done on purpose just to bill more hours.

One thing I'd add to the config-preservation idea: track cost per run alongside speed. I ran a multi-stage LLM pipeline in production (different models for different stages — a cheap/fast one for extraction, a stronger one for planning and a final quality check), and the parameter combo that scored best on quality sometimes cost 4-5x more per call than one that was ~90% as good. That gap only became visible once we logged cost per job with hard caps, not just latency/quality.

On the "why is this better" question: instead of trying to get the model to explain its own reasoning, we got more value out of a deterministic validator sitting outside the LLM entirely — plain rules that check the output's structure/constraints regardless of which model or params produced it. It won't explain the model's behavior, but it gives you a repeatable, non-probabilistic "is this run actually good" signal you can log and trend over time, even as the underlying models change.

"Unified playground" suggests comparison — but who's comparing? Researchers who need benchmarks, or product teams who need to choose a model? The researcher wants metrics and reproducibility. The product team wants speed and cost. Same playground, two different scoreboards.

I like the direction. Being able to compare the same model across different parameters feels more useful than just comparing different LLMs side by side.

Have you thought about helping users understand why one run is better than another for a given task, rather than leaving them to judge the outputs manually? That feels like it could be the real differentiator.