Why I built an AI Mechanic
I'm not a software engineer.
Like a lot of people today, AI made it possible for me to build things I never could have built a few years ago. Every day I'd ask AI to generate code, fix bugs, refactor features, and help me move the project forward.
But after a while I realized I had a new problem.
Every time AI fixed something, I found myself wondering:
Did it actually fix it... or did it just create six new problems somewhere else?
As someone without a traditional software engineering background, I didn't always know what "good" looked like. If something broke, my only option was usually to ask the same AI that built it to fix itself.
That felt like taking my car back to the manufacturer every time the check engine light came on and hoping they didn't accidentally create another problem.
That's where the idea for Jaikey came from.
Instead of asking AI to blindly keep patching itself, I wanted an AI mechanic.
Something that could inspect prompts, agents, workflows, and AI systems, identify weak points before they became bigger problems, explain what was wrong, and then repair them if I approved.
Over the last few weeks I've rebuilt the architecture from the ground up into a thin orchestrator with dedicated specialists for prompts, agents, workflows, multi-agent systems, and architecture. Both design and execution testing. Today I finished the first end-to-end regression suite, and for the first time the system behaved exactly the way I originally envisioned.
I'm curious how do other solo builders know when their app is ready? How do you test your own builds?
I liked the "AI mechanic" analogy. It captures something I've been feeling too.
AI makes it much easier to build, but it also makes it easier to introduce changes you don't fully understand. Sometimes everything still works, but you're no longer confident why it works.
For me, "ready" is usually less about finding zero bugs and more about reaching a point where I can predict how the app will behave when I change something.
Thank you, for me ready means people use it and everyone has the exact same complaint and hopefully its something easy to fix. If I get multiple issues all over the board then I got more work to do .
A score rising from 20 to 80 is encouraging, but it can still be self-consistency if the evaluator and repairer share the same model and rubric. I would add mutation tests with known defects, a frozen human-labeled set, repeated runs across seeds and model versions, and an independent judge for a sample of repairs. Report which evidence changed the score rather than only the number. Repairs should also run in a sandbox with a diff, rollback, and tests that prove the original behavior was not silently removed.
Solid points. You’re right that a 20 to 80 jump can still be self-consistency theater if the evaluator and repairer are effectively the same model + rubric. That’s exactly why Jaikey treats the score as secondary. Cause yes I am well aware Ai can toot its own horn or pat itself on the back.
What actually moves the needle for us:
-Mutation tests with known injected defects (we already run these on the agent-tester layer)
-Frozen human-labeled cases that never touch the repair loop
-Independent judge sampling on a subset of repairs (different model family when possible)
-Evidence-first reporting: every score change has to point to the specific failure modes or test cases that moved it
-Sandboxed execution with diff + rollback + regression tests that prove original behavior wasn’t silently deleted
We’re deliberately designing against the “looks better to itself” failure mode. The number is just a summary, l the evidence trail and the sandbox guarantees are the real product. Appreciate you calling this out. If you want to stress-test a specific artifact or see the mutation suite in action, happy to run one.
That is the right hierarchy. A stress test I’d want to see is a schema-valid semantic defect that survives ordinary unit tests, followed by a repair that improves the judge score but violates a frozen invariant elsewhere. Report detection rate, escape rate, regression rate, and repair cost. If the mutation suite and evidence trail catch that case, it is much stronger evidence than the 20-to-80 score by itself.
There’s an interesting tension in asking “when is it ready?” after finally seeing the system behave exactly as you envisioned.
Those sound similar, but they’re not necessarily the same milestone.
Exactly. Getting a system to behave the way you envisioned is one milestone. Knowing it'll keep behaving that way when the inputs, model, or environment change is a completely different one. That's where testing and verification become more important than the initial success.
Exactly — that gap between “it worked” and “it can be trusted to keep working” is often where the harder problems appear.
The car mechanic analogy actually clicked for me. The part about not trusting the same AI that introduced the bug to judge its own work feels very real if you're building without an engineering background.
It also made me think this is the kind of product that could naturally be getting a lot more organic traffic than it probably is today. When you're this early that can make a much bigger difference than people expect.
Curious what made you feel the regression suite was finally good enough to trust?
Thank you so much, Muhammad.
I think I started trusting it during an older version of Jaikey when it was primarily an evaluator. That evaluator is still part of the pipeline today. It performs evidence-based evaluations of both design and execution, scoring builds out of 100.
Early on, I deliberately tried to break it. I'd feed it poorly written prompts, broken agents, contradictory instructions, and even unsafe inputs. As it consistently identified the same problems I was seeing, my confidence in it started to grow.
The scoring behavior also made a big difference. One particularly bad prompt scored 20/100. After the first repair it only improved to 65/100. I ran the repaired version through it again, and it increased to 80/100. That gradual improvement actually made me trust it more.
One thing I've noticed about AI is that it's often overly confident. If a 20-point prompt had suddenly become a 95 after a single pass, I probably wouldn't have believed the result.
I've tested other AI QA tools the same way. Some seemed to hand out almost identical scores no matter what I submitted. I even gave one the vaguest instructions I could think of, and it still scored them 95/100. That was a red flag for me because it suggested the score wasn't really tied to the quality of the input.
I don't expect Jaikey to be perfect, but I do expect it to be honest about uncertainty and improvement. Seeing realistic scores that have to be earned, not inflated, has been one of the biggest reasons I've come to trust it.
That gradual scoring is actually what stood out to me too. A system that earns trust over multiple iterations feels much more credible than one claiming everything is 95/100 after a single pass.
Reading your post and your reply, I also noticed a few organic search opportunities that could help more builders discover Jaikey. If you're curious, feel free to reach out at mumar [at] growwithmumar [dot] online. Happy to share a few ideas—no pitch, just observations.
Thanks, I really appreciate that. And I'd definitely be interested in hearing your observations. SEO is actually one of the areas I'm still learning, so I'd love to see what stood out to you.
How do you currently handle your analytics and would you mind reading my indiehacker pots on how crawlers and bots are ruining analytics data?
https://www.indiehackers.com/post/i-just-discovered-my-analytics-numbers-are-mostly-fake-here-is-why-8197e3ff9d
Funny enough, I just started using PostHog, so I'm still learning what "normal" traffic even looks like. Your article definitely made me think about how much of what I'm seeing could be bots instead of actual users.
One thing I found interesting was your point about server logs being the ground truth versus client-side analytics. I hadn't really considered how much AI agents could skew the numbers until I read this.
I'm curious how are you distinguishing between legitimate AI crawlers (like search/indexing bots) and the resource-heavy agentic bots that are just burning CPU? That seems like one of the hardest parts to get right.
Enjoyed the article, that's good information I needed.
Really appreciate you reading! The PostHog angle is super interesting, PostHog is fantastic for product analytics, but because it relies on client-side events, it sits in that same blind spot where non-JS scrapers pass by completely invisible while headless bots trigger fake sessions.
To your question on distinguishing legitimate indexing bots (like official GPTBot/Googlebot) from resource-heavy agentic scrapers: it comes down to a three-tier check under the hood:
1. Reverse DNS (rDNS) Validation: Official crawlers from OpenAI, Anthropic, Google, and Perplexity either publish their IP ranges or pass PTR/forward DNS checks (e.g., verifying the IP actually belongs to .openai.com). If a request claims to be GPTBot coming from a random Hetzner or DigitalOcean IP, it gets flagged as a spoofer instantly.
2. TLS (JA4) Fingerprinting: Standard scrapers using Python (httpx/requests) or Node (puppeteer) often spoof their User-Agent string to look like a browser or official bot. But their TLS handshake cipher order (JA4 fingerprint) gives them away immediately—a Python client can't fake a real browser or official crawler's cipher signature.
3. Path Shape & Resource Cost: Official indexers just send simple GET requests to fetch your HTML pages. Rogue agentic scrapers perform deep interactions, hammering un-cached API routes (/api/search), looping through pagination, and running heavy DB queries.
So the platform categorizes legitimate indexing bots into a "Search/GEO" bucket (so you know your site is being indexed for AI answers), while flagging the rogue API-hammering bots as "Resource Drains" so you can rate-limit or block them.
Thanks again for the feedback—super helpful to hear how PostHog users are thinking about this!