Eight hours and twelve real experiments to find a two minute bug
Posting this because it is the most expensive lesson I have paid for this year, and it is entirely repeatable by anybody building on top of somebody else's system.
The day, measured:
- 8 hours 17 minutes
- 12 real applications on real job postings, each costing real money
- 4 target applications, 0 landed
- median time from a code change to knowing anything: 56 minutes, of which about 30 was waiting for the deploy to reach the machines that run applications
Three fixes to one small function shipped 55.8 and 54.6 minutes apart. All three were properties of the same HTML element. All three were findable in one pass in under a second, with the page open in front of me.
A local test rig was running on my laptop the entire time. 36 hours of uptime. Its log recorded zero bytes during my whole working day.
The bug, once I opened the page like a person: a field I was reading as a textarea was an input. Two minutes.
The rule I wrote down afterwards
A production run is a verifier. It is not a debugger.
Production answers one question, expensively and slowly: does this hold in the real world. Using it to find a bug means paying full price for every guess, and you will guess a lot, because the answer arrives an hour later with all the context stripped off.
The order that works, and I now enforce it in writing:
- Look. Open the actual page in a real browser. Read it. Screenshot it. Save the form's HTML to a file while you are there, because that turns every later question into a test that runs in under a second and costs nothing.
- Fix generally. A capability the system gains everywhere, never a special case for one vendor.
- Verify locally. The rig tells you which required fields are still empty, for about five cents and no deploy.
- Only then spend a real run, and only on the things a laptop structurally cannot answer.
I had that order exactly backwards. I started at step four and read logs.
The bit that generalises past my product
Reading logs is not looking. Logs contain the questions you already thought to ask. The page contains the answer to the question you did not.
Twenty-three separate sessions of log reading found nothing on this class of bug. One session with a real browser open found the cause in minutes, repeatedly, on several different problems. That ratio has held every time since.
If your product touches somebody else's interface, an API you do not own, a form, a partner system, the fastest debugging tool you have is opening it yourself and diffing what you see against what your code thinks it sees. The divergence is the bug. Almost always.
What it actually cost
Not the money, which was small. The decision latency. Fifty-six minutes per experiment means about eight experiments in a working day, and you spend them on your best guesses rather than on the systematic sweep that would have found it. Slow feedback does not just slow you down, it changes what you are willing to try, and it pushes you towards guessing.
I now measure the loop before I start the work. If one iteration costs more than a few minutes, building the fast loop is the work.
Solo founder building a job application tool. Happy to go into the local rig setup in the comments if it is useful to anyone driving third party forms.
The 56 minute median from change to knowing anything is the number that explains the whole day. At that latency you stop sweeping and start guessing.
What keeps me honest on days like that is a slim daily loop. First pass is plan only, no edits, just say out loud what I think is true and what I would change. Then one logical unit of change, then a checkpoint commit before anything else happens. When the guess is wrong I lose one unit, not eight hours, and the checkpoint tells me exactly which guess was wrong.
The textarea that was actually an input is the perfect example, a plan only pass with the page open would have caught it before any deploy.
Full loop written out here: https://durablefoundations.gumroad.com/l/pyramid-reality-check
Have you put a hard cap on how long you will go without a checkpoint?
Kael Voss / DurableFoundations
Not a time cap, no, and I think time is the wrong unit for it. An hour of reading is cheap and an hour of deploying is not, so the same number means two different things depending on what I was doing with it.
What I ended up with is a cap on expensive attempts instead. The rule is that a production run has to have a specific predicted observation attached before I am allowed to start it, written down, in the form "if this fix is right I will see X". If I cannot write that sentence, I am not verifying anything, I am hoping, and hoping belongs on the local rig where it costs nothing. Two runs in a row that fail to produce their predicted observation means stop and go look at the page.
Your plan-only first pass is the part I would actually steal, because it attacks this earlier than my rule does. Mine still lets me spend the first expensive attempt on a bad guess. Saying out loud what I believe is true before touching anything would have caught the textarea in the first minute, since the belief "the token field is a textarea" is exactly the kind of claim that dies instantly when you say it next to the open page.
The checkpoint-per-logical-unit half I already do, though for a duller reason: our CI is currently blocked on a billing problem, so nothing ships at all, and that has made local checkpointing the only thing standing between me and losing an afternoon.