LLM Knights
The Unified Playground for LLM Testing
Now, you can add models against each other. The next step is to add more providers and more models. Which providers would you like to have next (currently, I have only OpenAI)?
For this project, I have to provide a visual UI for users to change settings before starting testing. I found there is no any consistency not only among different AI providers (like OpenAI, Anthropic, and others), but among the models of the same provider.
For example, many modern models by OpenAI don't allow the temperature setting (and it's a pity, I found it very useful), and if you accidentally pass it to the API request, it will fire the exception.
Another black hole is reasoning parameters. Some models support reasoning effort, some don't, and those that do, they all have different set of allowed values.
For example, GPT-5.6 Sol supports a bunch of values: none, low, medium, high, xhigh, and max. But o3-mini allows only low, medium, and high.
The same with reasoning mode.
But the main problem is there NO f**g documentation saying directly and explicitly which model supports which sets of parameters with which set of values.
This is what happened before I discovered all this mess:
I've found several publicly available JSONs describing models but they lack a lot of data.
-
1
This is painfully accurate. I ran into the exact same issue while building an AI-powered app - temperature randomly breaking, reasoning params varying between models, and zero clear docs on what actually works where.
The worst part is you don't find out until runtime. Your app works perfectly with one model, then you switch to another and suddenly everything throws exceptions.
I ended up building a small internal wrapper that normalizes these inconsistencies across OpenAI/Anthropic/Gemini/etc. It's been such a time-saver that we productized it as ModelBridge - one OpenAI-compatible API that handles the param mapping so you don't have to maintain a giant switch statement for every model.
Not here to sell though - just saying I feel your pain. What's your use case? Are you building an app that needs to support multiple models, or are you mostly stuck on one provider?
It was not easy to decide on pricing.
Yeah, usually, with all my previous SaaS apps (most of them flopped, FYI :) it was not hard: I just used my own boilerplate with the Stripe integration, create monthly pricings, trial or free period, and that was it.
With LLMKnights, it's not that easy.
First of all, I don't want to have any monthly plans. Why? Because, personally, as an end user, I hate them. I tried many LLM providers and never subscribed to any monthly payment. Most of them are more wise: they allow you to fund your account as few as $5 and use them whenever you want.
So, I decided the same: I will charge the users based on usage and they may decide on how many to pay.
Another question was exactly how. I know, most providers just add 5,5% to the original provider's price (OpenRouter does so and many others) but. My product is not an LLM provider. It's the testing platform so I don't expect high volume of requests. And the functionality is kinda unique (to some extend) so it's not that easy to move to a competitor doing the same.
But having a fixed price doesn't make sense either. So, I decided to "invent" credits. One credit in our terms is how much you spend on making a simple not big call to an average provider. That's it. That said, making such a call to Deepseek-4-flash may cost you 1 credit but to Deepseek-4-pro - 2 or more.
Still, 200 free credits is very enough to try the service and decide if it's worth to use the tool further.
To get more feedback, I also give 5,000 free credits to anyone who leave their email before launching and will sign up when we launch. I think I may launch in a couple of weeks.
So... if you want to get those 5K credits... please leave your email here: https://llmknights.com
I know that it's not useful for many tests (like "Generate me a video") but I also know, it's very important for many other tests.
If you have a chatbot answering your users' questions, this could be critical, but I bet none of the chatbot providers give you an opportunity to run tests side-by-side to see how different models (or the same model, but with different params) cope with conversations.
Here we are.
And here is one of tests (single-model test set, with only param set, kinda a smoke test to see it's working):
I'm so crazy satisfied with the result! I was not hard to implement but required a lot of attention which I usually lack haha.
Tomorrow, will work on the landing page. It's so empty and boring :-\
Even before building my app, I had many ideas of AI-based products and the infinite unsolvable problem was comparing LLMs pricings. I found several pages/websites on the Internet but they all lack either models or filtering options.
So, I decided to make it on my own. Found several free sources on the Internet, then built a searchable / filterable table. So, now it has 300+ LLMs to compare. Please note: it allows to compare LLMs against each other, not the providers (this is coming).
https://llmknights.com/llmprices
I hope you enjoy and let me know if you lack something :)
-
1
Per-token sticker price is the wrong unit of account, and it is why every comparison table quietly misleads. At SocialPost.ai our cheapest model per token was not our cheapest per finished output, because verbosity and retry rates differ so much between models. Since you are already building a testing playground, publishing cost per completed task from your own test runs would make this table something nobody else has.
-
1
This hits close to home. I'm building a tool that turns podcast transcripts into visual content using AI. The generation cost per piece is real, and it makes pricing the product much harder than I expected.
-
1
Yes, I can see it. Another thing that is probably a challenge is choosing a model that fits your needs and shows the best ration price/quality. This is why I created my tool - you will able to test different models, compare the result and metrics like cost, latency, token amount.
-
-
1
Seems right now that routers are big business, they're in demand as a tool and as an acquisition target.
I wonder if there's some obvious tie-in here for you to monetise.
https://www.justwatch.com/ offers availability of films across different providers - could you have a view like that? Or another analogy skyscanner.net-
1
Hi, thanks for your question but unfortunately I don't get what you mean. I'm planning to add different providers information as well as make them available for testing on our platform. Is that what you mean?
-
-
1
LLM cost is definitely one of the biggest challenges for AI products.
I think the key question is whether the AI experience creates enough value that users are willing to pay more than the inference cost.
-
1
Hi, I think every providers has some features that make them valuable more than paying to providers directly.
-
1
That's a good point. I think the challenge is becoming "which model/tool should I choose for this specific business use case". There are too many options now. Do you use any tools to compare AI products or track new AI startups?
-
-
-
1
the “sites either miss models or can’t filter” frustration is painfully real. Shipping a searchable table of 300+ LLMs with price, context, and modality filters is the version of “I’ll just build it myself” that actually helps people.
Provider compare coming is the right next layer — model-level is how you shop; provider-level is how you commit.
What’s the first filter or column people ask for that you still don’t have — latency, rate limits, or something messier like “same model across APIs”?
-
1
Thanks!
What’s the first filter or column people ask for that you still don’t have — latency, rate limits, or something messier like “same model across APIs”?
If your question is about the tool itself, we already have all of them. Multimodel testing is coming soon.
For the table, not sure whether latency and such can be integrated at all there. Thoughts?
-
1
latency in a static price table is tough: it swings by region, time of day, and provider route, so a fixed column goes stale and can mislead.
I'd keep price/context as the core, then a measured badge linking to p50/p95 elsewhere — or an optional filter with a clear caveat that numbers are snapshots, not SLAs.
When multimodel testing lands, are you ranking same-prompt quality first, or cost/latency across providers of the same model?
-
1
Thanks for sharing your thoughts, definitely will think about it!
There are 2 modes: single and multi-model. In the single model mode, you create param sets (different temperature, reasoning effort and such) and run the test with the same system prompt and user input against them.
In the multi-model mode, you select different models (with their own parameters) and run the same system prompt + user input against them. In both cases an each individual test comes with latentcy, input/output token numbers, and calculated price (based on an original provider's price). Later, I may provide the calculation for other providers (based on the same token usage).
-
-
-
-
1
great, think it's helpful
-
1
This is actually a problem a lot of AI builders run into.
LLM pricing changes so quickly that comparing models becomes almost a product decision by itself.
The filtering aspect is probably the most valuable part — founders usually don’t just need the cheapest model, they need the best tradeoff between cost, latency, quality, and use case.
Would be interesting to see filters around things like:
best models for agents
coding
RAG/search
long context
structured outputs
Great resource. Keeping this updated is probably the hardest part.
-
1
Hi, thanks for your comment!
What you suggest is already in my mind! The whole point of the service is allow people to test LLMs and of course, we will do testing too and will publish the results of our testing. So, stay tuned!
-
1
This is a real gap, glad someone built it. We run multi-provider AI features and the pricing comparison part was way more annoying than it should've been, every provider formats it differently and cached/prompt-caching pricing especially gets left out of most comparison pages, like the other commenter mentioned.
Would be genuinely useful if "LLM Knights" eventually tracked cache pricing too, since that's often the difference between a model being cheap or expensive in practice, not just the sticker price per token.
-
2
Thanks for your suggestion!
Noted!
-
-
1
The column I would want is cached input price. Anything with a stable system prompt pays the cache-read rate on most calls, and that can be a fraction of base input, so two models sitting next to each other in your table can be far apart on a real bill. Do the free sources you are pulling from expose it?
-
1
Thanks for your suggestion, we definitely will provide this information soon.
-
-
1
the only person worth competing is the person you were yesterday
-
1
I agree 100%. Thanks for your comment.
-
-
1
llm pricing comparison is one of those things you assume exists properly until you actually need it and every page is missing half the models. building it because you kept hitting the wall yourself is usually how the useful ones start.
-
1
Exactly. It's still not perfect and not full, but I just tried to water to see if somebody else except me needs it :)
-
Today, I finished this part of coding: now, I can select a model and then add test runs to test this model against different parameters.
For example, I can change the temperature (if this model supports it), or level of verbosity or reasoning efforts.
Now, it looks this:
It still lacks some important features like the parameters that the tests were run with. But I will add it, soon.
-
2
The feature that would make this a daily tool for me is saved test suites I can re-run when a provider silently updates a model. We run LLMs in production at SocialPost.ai and the pain isn't finding the best parameters once, it's catching quality drift after we've shipped. Logging params per run is table stakes; re-runnable suites with diffs against the last run is the killer feature.
-
1
Wow, this is an interesting feature. So, by re-running do you mean just rerun the same combination of parameters, system prompt, and user input? Or are you talking about a multi-message conversations?
If the former one, it's already implemented! You can always click the re-run button to start a new test run, and it will grab the system prompt, user input, and all the param sets (which can contain different params for a single-model test set, or different models for multi-model test sets).
This is how it looks now:
For the conversational test sets, probably creating a copy of it could be a solution.
Hope, I answered your question :)
-
-
1
Every time I assign a task, it’s never fully completed. Something always gets skipped or ignored, even when I provide a detailed spec. I have to keep adding things later, which means more costs. It’s frustrating—it honestly feels like it's done on purpose just to bill more hours.
-
1
Are you talking about your experience with what?
-
-
1
One thing I'd add to the config-preservation idea: track cost per run alongside speed. I ran a multi-stage LLM pipeline in production (different models for different stages — a cheap/fast one for extraction, a stronger one for planning and a final quality check), and the parameter combo that scored best on quality sometimes cost 4-5x more per call than one that was ~90% as good. That gap only became visible once we logged cost per job with hard caps, not just latency/quality.
On the "why is this better" question: instead of trying to get the model to explain its own reasoning, we got more value out of a deterministic validator sitting outside the LLM entirely — plain rules that check the output's structure/constraints regardless of which model or params produced it. It won't explain the model's behavior, but it gives you a repeatable, non-probabilistic "is this run actually good" signal you can log and trend over time, even as the underlying models change.
-
1
Thanks for your suggestions!
We already show the cost, see here:
As for validator, this is great idea, and I have it in my mind, too. I'm considering to implement it in 2 ways:
just with plain rules (like length of text, presence of specific words, json schema validation, and so on)
AI-based validator that for more complicated validation: like check the facts, or presence of the specific information, profanity check etc.
So, do you mean if the model changes, you will see it by running the same test again the validator, right?
-
-
1
"Unified playground" suggests comparison — but who's comparing? Researchers who need benchmarks, or product teams who need to choose a model? The researcher wants metrics and reproducibility. The product team wants speed and cost. Same playground, two different scoreboards.
-
1
Thanks for your question! Mostly, it's for founders and developers - which you call "product team", for choosing a right model. Speed is already among the metrics (you can see it on the screenshot), and cost is coming. Researchers can use it too, we can add more metrics in the future.
-
1
“Got it — makes sense focusing on product teams first.
One thing we’ve seen: even when speed + cost are clear, teams often struggle to decide because the ‘best’ model shifts by use case.
Would be interesting to test how different builders actually choose — happy to share what that surfaces.”
-
1
Totally! The sharing user feedback is already on the roadmap (with the test cases if they shared them too). We also will do tests different models x different test cases to demo what someone can even do with modern AI, should be fun!
-
-
-
-
1
I like the direction. Being able to compare the same model across different parameters feels more useful than just comparing different LLMs side by side.
Have you thought about helping users understand why one run is better than another for a given task, rather than leaving them to judge the outputs manually? That feels like it could be the real differentiator.
-
1
Hi, thanks for the question!
Interesting idea, but does someone really need to know why the result is better if they can just reuse the set of paramters?
Another thing is that I suppose it's not always easy to explain, especially with AI which is probabilistic and "deep mind" why a specific set of params gives the better results than others. But may think this direction, too.
-
1
That’s a fair point. I wasn’t thinking so much about explaining the model’s reasoning, but helping users build confidence in the result. If I keep coming back to the same task months later, I’d find it useful to know why I saved this configuration or what kind of tasks it consistently performs well on, even if the explanation comes from patterns in previous test runs rather than the model itself. Have you thought about preserving that context alongside the results?
-
1
Hmmm can you elaborate a bit more on "context"?
Right now, I store everything that can be useful: messages, prompts, params (like temperature, reasoning efforts etc.), latency, cost at the moment. Do you mean something else?
If you are talking about some user's comments, I will implement it soon as feedback - from it, you can see later, what you liked and how.
Btw, as an example of situation when it's hard to explain the result, look at this: today I was testing the params and reasoning efforts. The user input:
If I have 2 apples and John has 3 apples, and John gives me one apple, how many apples are there?Model: o3
Test 1
reasoning {"effort":"low"}
Before any apples are exchanged, the total number of apples is:
• You: 2
• John: 3 Total = 2 + 3 = 5 apples John gives you 1 apple. This only moves an apple from John to you; it doesn’t create or destroy any apples.
After the exchange:
• You: 2 + 1 = 3
• John: 3 − 1 = 2 Total = 3 + 2 = 5 apples So, there are still 5 apples in all.
Latency 2.27 s
Input tokens 36 - 0.000072
Output tokens 198 - 0.001584
Total cost 0.001656
Test 2
reasoning {"effort":"high"}
There are still 5 apples in total. After John gives you one apple, you have 3 apples and John has 2, but the overall number of apples hasn’t changed.
Latency 2.06 s
Input tokens 36 - 0.000072
Output tokens 251 - 0.002008
Total cost 0.00208
Honestly, I can't explain this result. The only guess is the temperature is not 0, may be 1, and therefore, this result is not related with reasoning effort, but rather with randomness.
-
1
was thinking less about explaining why the model generated a specific answer and more about explaining why I saved that experiment.
For example, imagine I save a configuration with a note like:
“Works well for short customer support replies.”
or
“Reliable for structured JSON outputs, but slower and more expensive.”
Months later, I don’t have to remember why I bookmarked it—I immediately know when to reuse it. That kind of context comes from my own testing patterns rather than the model itself.
Your planned feedback feature actually sounds like it could evolve into something very close to this.
-
1
Ah, this makes sense. But I guess, it's already on the road map as the feedback feature. You will be able not only rate an every single test result by different criteria, but also leave the note to older yourlself!
Let me know if you mean something else :)
-
1
That sounds like exactly what I had in mind. I like that the notes and ratings become part of the experiment itself rather than something users have to remember separately. Over time, I could see that becoming a personal knowledge base of what works for different tasks. Looking forward to seeing how it evolves.
By the way, are you active on X, Discord, or Telegram? I’d love to follow along as you keep building.
-
1
Sure, you can find me here https://x.com/SeaCatWiz I'd be glad to connect!
Also, don't forget to leave your email because... Finally, decided on pricing and offer a free gift to all the early birds (read to know how) - Indie Hackers
-
-
-
-
-
-
-
Comment