devdigest.io
AI-curated daily tech newsletter
-
1
The fact that the metric was technically correct but still misleading is a nasty one. Especially when the label makes the distinction invisible.
-
1
That is exactly the part that got me. A wrong number you argue with. A number that is correct under a definition you have forgotten is one you act on.
It caught me again today in a different shape. A test in my suite passed while making a live API call and cancelling a real subscription. Green run, because the assertion was satisfied by the side effect it should never have caused. Same structure as the metric: technically true, and answering a different question than its label implied.
What I have taken from both is that the fix is almost never in the mechanism. The filter was correct and I kept it. The test was correct about what it asserted. What was missing both times was something making the gap between the label and the question visible at the moment someone reads it.
-
1
That’s a pretty strong pattern — the number can be correct while the decision it drives is wrong. The live API example makes that especially clear. Curious whether you’re thinking about this as a broader product principle now, rather than just a metric/testing issue?
-
1
Yes, and it happened a third time since I wrote this.
Today I reviewed a fix over a five day window. The target said "roughly 10 to 25 stories demoted per day". Two of the five days came in at 4 and 5, which reads as the fix underperforming. The eligible pool on those days was 27 and 40 stories instead of the usual 200. As a rate it was flat across all five days.
The count was correct. The conclusion I would have drawn from it was wrong, for the same reason as the filter: the label carried an assumption the number did not.
So the principle I work to is narrower than a product philosophy. When a number is going to drive a decision, write the denominator and the assumption beside it, in the tool, where the person reading it will see them. Not in documentation, which nobody opens at the moment they act.
-
1
That’s a very useful distinction — especially the idea of putting the denominator and assumption where the decision happens. I’d be interested in continuing this conversation. What’s the best email to reach you at?
-
-
-
-
-
1
Hi! I tested the signup flow and eventually got access to the preferences and past-digests pages through the email links. Everything appears to be active, but as a new user I found the flow confusing:
After confirming the trial, the homepage still showed “Start free 14-day trial”
The “See a real issue” page remained blurred and still asked me to sign up
“My Past Digests” required another email link, and the archive was empty because the first digest had not arrived yet.
I initially thought my subscription had not been recognized. It may help to clarify that the sample remains gated, that account access is email-link based, and when the first digest will arrive.
Also, I couldn't find a way to unsubscribe from the newsletter; perhaps it's done via email, but the process remains confusing.
-
1
Hey Ian, firstly I really appreciate you actually testing the signup flow end to end and writing this up, this is exactly the kind of feedback that's hard to get otherwise.
Went through every point:
Homepage still saying "Start free trial" after confirming - fixed. Confirming now takes you straight to your own preferences page (topics, story count, etc. already showing what you picked) with a "you're confirmed" banner, instead of dumping you back on the marketing homepage with nothing to prove it registered.
/sample staying gated and asking you to sign up again - fixed, for the normal path: from your own preferences or archive page, the "See a real issue" link now recognizes you and shows the real thing with no gate. Worth being upfront that there's no login/account system here by design (no tracking, no cookies) - so that recognition only travels through that specific link. Click something else first (like the logo) and it reverts to the anonymous view, since there's genuinely nothing stored anywhere to remember you by. Decided that's an acceptable tradeoff to keep the no-cookies posture rather than build real accounts for it.
Empty archive, no explanation-fixed. It now says plainly that you just signed up and the first digest hasn't landed yet, instead of just looking blank.
No way to unsubscribe - you were right, there genuinely wasn't one anywhere on the site before your first email arrived. Added a direct Unsubscribe link on the preferences page.
My digest is aimed to fire at 7am (ish) of the subscribers time as it aims to be the "morning coffee&news" experience.
Your core point - "I thought my subscription hadn't been recognized" was the real bug underneath all of these, so thanks for naming it that clearly. Let me know if any of this is still confusing once you look again.
(bumped a few extra trial days as a little token of appreciation) Cheers!
-
-
1
I'm curious what would convince you this is solving a fundamentally different problem from every other AI news digest, rather than just presenting similar information in a different format.
-
1
Honestly, the underlying problem (too much AI/tech news) is the same one everyone's solving, so I won't pretend otherwise.
Where I think it differs is the mechanism, not just the format: it's not a static algorithm applied the same way to everyone - every downvote with a reason (wrong topic / wrong source / too shallow) actually reweights what you see next, so two subscribers with the same topics can end up with visibly different picks after a couple weeks. And every story shows why it was picked, so you can actually audit that instead of trusting a black box.
What would convince you : try it for a week or two, downvote a few things with a reason, and see if the next day's picks actually shift.
-
1
Appreciate the context.
Would be good to continue the conversation outside the thread.
What's the best email to reach you on?
-
-
3 Comments
Partly, and the split is the honest bit.
What is actually tested now is the code that computes the metrics. The open-day guard has clock-pinned tests, and it is not bolted onto one metric - ten call sites route through it, so a metric cannot quietly opt out. Baselines are data rather than prose: each ledger entry records the pre-ship number before the change ships, and every figure shown is recomputed from the raw daily records rather than stored, so a number cannot drift away from the thing that produced it.
What is still only discipline is the definitions themselves. Nothing checks that a target contradicts a bound recorded four lines above it in the same entry. Nothing checks that a baseline and its live metric count the same unit. Those are the two that actually bit, and both are still a human rule I can forget.
The fix I keep circling is making the unit a declared field on both the baseline and the metric, so a mismatch is a config error at ship time instead of a discovery on review day. Not built yet.