A 5-minute fix that became a 3-hour debug session
Last Wednesday at 16:55 IDT, Sentry pinged me about a Resend 403 on a welcome email that did not reach its destination. It should have been a 5-minute env-var fix, but it turned into a 3-hour debug session against three different tools, each of which broke in a way I'd never seen.
Tool #1 was the backend coding agent: I asked for a retry script that reads DATABASE_URL from env. The resulting script required setting the envvars temporarily in the terminal, but since I was stressed out I pasted the literal envvars directly in the scripts, which was later committed to Github. Caught the hardcoded prod password 10 minutes after push, so we force-reset the branch and rotated both creds.
Tool #2 was Supabase: I rotated the password 3 times across ~60 minutes, every attempt failed against the Session Pooler with 28P01. Eventually tried the Transaction Pooler at port 6543 with the same credentials, which connected fine. The Session Pooler was probably caching the old auth for ~30 minutes after every reset. Same password, different pool, one fails and one works.
Tool #3 was Chrome: I had a browser agent helping with the rotation clicks, and the design intent was that the password never enters the agent's context: the agent clicks the Supabase "Copy password" button, the clipboard carries the value, and I do the pasting into Railway. However, the clipboard never really updated. Chrome locks clipboard writes behind real user activation, and CDP-driven synthetic clicks don't always satisfy that, so I was pasting stale buffer the whole time.
Every single one of these looked like a normal auth error at the Railway end. The lesson: when 3 different tools all silently fail in a row, extinguish the fires first, then sit down to reflect. Dwelling on placing the blame just slows you down.
Action items I left with: install GitLeaks across all sub-repos, promote pre-push grep verification into the default agent teammate brief, test both Supabase pooler ports before re-rotating, real-mouse click + Notepad-verify for any credential rotation driven through the browser.
Postscript, because the deepest fix only landed yesterday, as I was writing this. A follow-up dev rotation still cost me another ~30 minutes of database downtime: resetting the shared postgres superuser password in place breaks every existing connection at once, and then the pooler cache lag sits on top of that. Now, the app connects through a dedicated least-privilege role instead of the superuser, and rotations are a two-role overlap: create role B with a fresh password, cut the connection string over while role A stays valid, then retire A. And the result? The dry run had zero seconds of downtime. The next rotation should be a non-event, which is the whole point.
What's your heuristic for the moment you stop suspecting yourself and start suspecting the tool?
My rule is usually: if the same symptom survives three independent explanations, I stop assuming I'm the only variable.
What stood out more to me, though, is the process change at the end. Most of the post reads like debugging incidents, but the dedicated role + overlap rotation is really a decision about reducing future operational risk, not fixing a one-off outage.
That's the difference between solving the failure and removing the class of failures.
Worth separating those two because founders often celebrate the fix and miss the process change that actually matters.
Happy to put the tighter version in writing if useful. This feels like one of those lessons that's more valuable as an operating principle than a debugging story, and I'd rather not crowd the thread with it.