PromptPerf

Back in May, I launched PromptPerf on Product Hunt. Got 161 upvotes and some brutally honest feedback:

- "Sadly no Gemini"

- "API key friction"

- "Need multi-model comparison" — multiple users

I spent the last 7 months building exactly what they asked for. I had been informed that there is a 6 months wait period from ProductHunt before a relaunch hence I decided to make small changes over the period and monitor user behaviours.

What changed:

❌ Before: 3 OpenAI models only

✅ Now: 100+ models (GPT-4o, Claude, Gemini, Llama, DeepSeek)

❌ Before: Required API key to try

✅ Now: Works without signup — free trial

❌ Before: Test one model at a time

✅ Now: Compare 5 models side-by-side

What I learned:

1. Your first launch is research, not revenue

2. The comments matter more than upvotes

3. Building what users ask for > building what you think is cool

We're live on Product Hunt today. Would love your feedback or just an upvote if you think this is useful. Product Launch

I launched 2 Months ago, Product Hunt put me on #7 for the day over 100+ upvotes insane for someone who was just building something to solve my own troubles.

I found evaluation of prompts tiring job, run the same prompt across mulitple models and keep tweaking it everytime to get the optimal results. From a developer lens I saw this as there has to be a way to do test cases for prompts, and test it against multiple models to check reliability and accuracy. As everytime a model gets deprecated we need to make the switch quick.

So I built promptperf.dev

While building the product I might have hung onto the thought of this product will make me my first million haha. On Launch Day seeing 100+ upvotes I started to see dreams of getting acquired for couple hundred thousand atleast lol.

Soon enough, I started to note that users were just signing up not sure how to use the app. So I iterated, and iterated again my metric for success became from signups to number of tests/evaluations run, so the goal shifted from getting people onboard to getting them to run the first testcase.

I now have 26 users who have ran their first test case. But only 7 have done it more than once. Its clear, users are not coming back, use it and dont need to keep coming back.

What this means, I should not target monthly pricing as clearly I'll be robbing the customer if they only have to use the app every so often. I've currently got Lifetime pricing at $49 which I think is fair given I need to calculate possibilities of the runs a user might make and keep the site up and running. I might need to implement $10, 10 runs plan or so I guess but since theres only been 7 actual users who decided to come back, Im putting it on the back burner as seems like I need to do a better job at explaining the problem to the users or clearly need to target more specific groups.

Really keen on some advice. (I tried Apollo for cold emails -> But you need to warm up your accounts before emails get sent out hence still not executed that step)

1 month, 62 signups, and a lot of clarity

If you’ve found value in this journey, it’d mean a lot to get your support on the v3 launch:
👉 https://www.uneed.best/tool/promptperf

When I first set out to build PromptPerf, the big vision was obvious:
A tool to evaluate prompts across multiple models GPT-4, Claude, Gemini, and more.
That was always the goal. But I didn’t start there.

Instead, I took a different path one that felt slower, but turned out to be much smarter.

Three weeks after the waitlist landing page went live, I shipped the first working version.
It was barebones, but functional:
→ Enter your API key
→ Download the template
→ Manually write your test cases
→ Upload the CSV
→ Evaluate on one model

And guess what? Almost nobody used it.

That was the first lesson: friction kills usage.

So I shifted focus not to build more features, but to remove barriers.

🔹 v1: CSV upload + user’s own API key
🔹 v2: Removed API key requirement
🔹 v2.5: Added test case templates, onboarding flows, and a browser-based editor. this meant, user could evaluate within seconds

These fast, lightweight iterations taught me more than any feature request:
→ The problem wasn’t depth, it was setup fatigue
→ Most users didn’t want more options, they wanted momentum and ease to use

Show the problem to the user. They might not know what problem you are solving

That changed everything.

By the time v2.5 rolled out, I saw something I hadn’t before:
8 out of 10 new users were running at least 1–2 evaluations.
That was the first real signal.
Not just signups. Not just traffic.
Usage.

Only then did I ship the “hero feature”:
✅ Multimodel Evaluation — side-by-side comparisons across GPT-4, Claude, Gemini
✅ Rebuilt the dashboard
✅ Redesigned the export logic to cleanly handle multi-model output

Looking back, the big takeaway is clear:
Shipping the complex feature first would’ve hidden the real problems.
By delaying it, I uncovered what actually stopped users from completing the flow.

Now they’re not just signing up — they’re staying.

Still building. Still learning.
If you’ve found value in this journey, it’d mean a lot to get your support on the v3 launch:
👉 https://www.uneed.best/tool/promptperf

I had shipped a MVP version of my product https://promptperf.dev I launched prompt testing with users API Key and only upload csv/json for the test cases.

Then I pivoted and made it so users can enter test cases on the app and also do bulk upload AND BIG PIVOT was to remove user API Key and allowed direct usage so I bear the API costs.

Now Im launching multi model runs. Heres a sneak peak of the dashboard. Please provide feedback if this looks good.

https://devhunt.org/tool/promptperf

Support with upvote.

I launched on PH 10 days ago and got 50 signups. But this excitement soon hit reality when I found out no one was using the app. I had set a analytic metric on number of prompt evaluations ran.

That number didn't move. No one who signed-up was using the app and the reason was staring at me. Users required to enter their API Key. I took the concept from typingmind who generates over 6 figures annually and requires your API key.

So I took a big step allow free users to use my API and did some number crunching and also allowed lifetime users to use my API. Now users dont need to enter their API keys.

Another change was created a how to guide using supademo and also added a banner on the page that quickly adds test prompts and makes it easy for the users to hit just one button to run the evaluations.

I am launching again but this time on devhunt to check if these changes will get me my first evaluation run by a user.

please support and upvote on devhunt to help get more users.

Thank you everyone for the support. I have received nearly 40 signups and 1 paid user which is massive for me as I am still on the early stages of validating the product. So thank you everyone.

Next steps: Even though the signups are coming in I am tracking the usage of the app and I dont see many users running the evaluations and I need help. How should I get you to try and test the product.

My current thoughts are:
- User using their API keys means friction and would go away from the platform as can't test it immediately so perhaps allow free trial with my API key which involves unlimited runs with 3 test cases.

- Create a onboarding guide? Like the ones of enterprise softwares that says "Click here" "Next Steps": What tool can I use for this?

For both the options above will still need to inform them about the updates and hope they will signin again.

- Reach out to all 40 users for a 1-1 15 mins call and show them the product. Assuming 30% respond that 12 calls booked.

Do you have any suggestions? Keen on feedback. This is critical as I need to solve these issues before building next features i.e (Adding more models and multi model runs).

Please help me build a product that solves your problems. https://promptperf.dev

Im new to the game and dont have a huge marketing or product strategy team hence each one of you is my strategy team :)

So I launched on PH yesterday and had emailed the 53 early adopters the day before to try the product out and in return for being early adopters once they signup I will upgrade their accounts to the paid version.

PH launch was scheduled and went live my 5pm and was just shocked on how quickly the upvotes started to come. I got almost 8-10 people reach out via Linkedin for a paid collab which was abit disappointing knowing others can easily use this to show their products higher on the leaderboard. Please upvote if you haven't as this is the only thing I can ask haha dont have $$ to pay to these influencers

I ended up at #7. You can checkout my launch here: https://www.producthunt.com/posts/promptperf

So I figured since I launched on Friday, theres was less competition as everyone is rushing for a Monday launch so they can get the top weekly feature.
I then woke up the next day and didnt even bother checking stripe. I saw 19 users and eventually 26 as of writing this post but after lunch I decided to download the stripe app on my phone and I launch the app to $$. I just was so confused so open my laptop. I kept refreshing the DB and Stripe multiple times to see if its the test account or some glitch. It wasn't I made $$ from something I built.

https://www.producthunt.com/posts/promptperf


I launched on PH and in 10 mins I got 9 profiles messaging me for a promotion ranging from $50 - $1000 some even with guarantee of 100 upvotes.

I have $0 ad spend planned so had to decline all of them, but makes you wonder how many PH products are completely organic

I just couldn't go to bed thinking about not being able to launch just because the payment button doesn't work.

So I launched with this promptperf.dev

This will allow users to signup and use the free testing which allows 3 test cases per run unlimited runs. So this is a good starting point. So i shipped it.

Its 12:30 now and just sent out the emails to all the early adopters, I will be manually upgrading each of these to Paid lifetime access for free forever. This is just a way to say thank you to all who supported by signing up on the waitlist.

Next steps:
- Get the payment button to work
- Gather feedback from early users
- Implement more AI models and increase price
- Only follow the roadmap to build a new feature for every new 100 users to ensure we get enough feedback and traction before building more features.

I wanted to launch today but its 11:38pm here and the problem is that the payment system doesnt work.

If the payment goes through then it doesn't redirect the user back with success.

And now if the user goes to pay it will cause an error

Been on this for 6 hours after completing the day job. Just doesn't work

I am going to put a hold on this till tomorrow and fix it after a sleep.