devdigest.io

The last two things I posted here were both measurement defects of mine: a filter that made a live source look dead, and a directional error that manufactured a trend out of one incomplete day. I have now run enough of these reviews to say that was not a coincidence. It is the pattern.

Some background. I build devdigest, a daily tech digest that shows you why it picked each story. Every change that changes behaviour goes into a ledger with a written prediction attached, and a review date is set the day it ships, usually five days out. On that date I have to come back and read the real numbers against that prediction. Not a vibe check - the entry records the pre-ship baseline so I am comparing against something I committed to before I knew the answer.

Twelve changes have been through it. None has been reverted.

You could read that as the review being toothless. I would have, six weeks ago. But eight of those twelve reviews did find something broken, and in every one of the eight it was the measurement rather than the change.

Three of them, concretely:

A target that was impossible when it was written. One fix's own entry contained an A/B bounding its effect at roughly 65 to 70 percent fill. The target sitting a few lines below demanded 91.7 to 97.6. The fix hit its bound and "failed". The rule I added afterwards: when an entry contains an experiment that bounds the effect, the target may not exceed the bound.

A baseline in the wrong unit. The recorded baseline counted distinct stories. The live metric counted slots in a digest. Same words, different denominators, and the review read wrong in the direction that flattered me - which is the direction you are least likely to go back and question.

A metric that moved when my subscribers did. This one is the worst, because both readings were defensible. The metric pooled per-slot shares across whoever received a digest that day, so it moves when the subscriber mix moves and nothing in the pipeline has. Two subscribers stopped receiving mid-window. The same five days read +1.45 pooled and +4.26 with them dropped. Only the per-subscriber version is composition-free, and it was the one that showed the fix had actually worked - on 6 of 6 subscribers.

I nearly recorded that last verdict backwards. "Failed target, not failed fix" was already drafted.

The generalisable bit, if there is one: we test code and we do not test measurements, and a broken measurement is far more dangerous than a broken function, because it does not throw. It returns a number, and the number looks exactly like an answer.

devdigest went live on Product Hunt yesterday if you want to look at the thing itself: https://www.producthunt.com/products/devdigest-io

And the product: https://devdigest.io/?ref=indiehackers

Last time I posted here it was about a filter that made a live source look dead. Same shape of problem again, one layer up. This time the instrument was fine and the calendar was the bug.

I build devdigest, a daily tech digest. Every change that changes behaviour gets reviewed against real data five days later, and I write the numbers down before I let myself have an opinion about them.

This week's series, the share of each digest landing in the four categories a fix I had just shipped was meant to feed:

69.2, 67.6, 60.0, 53.0

Four days, straight down. I had the write-up half composed in my head.

Then I noticed the last number was wrong.

Subscribers choose when their digest arrives. The first goes out just after five in the morning, the last at midday. My measurement treated the current day as a finished one, so reading it in the morning meant measuring a day that had not happened yet. The real figure was 53.9, not 53.0.

Nine tenths of a point. Not exciting. Here is why it is worth writing down.

The error is directional, not random. The late digests skew one way, so an early reading is always wrong in the same direction. And it is always wrong on the newest point, which is exactly where you look when you are deciding whether something is trending.

A random error adds noise you can see. A directional error on the most recent point manufactures a slope.

Day five then came in at 61.9 and did not continue the decline.

My first instinct was to write that up as the decline never having been real. That instinct is wrong, and it is wrong in the same way as the one it replaced. Four points looked like a trend. Five points look like a recovery. Neither is something five points can tell you, and day six could resume the fall.

What I can say is narrower. The four-day drop was 15.3 points. The largest four-day drop across the twenty-two days before it was 8.7. So something real happened, and waving it away as noise would have been the mirror image of the mistake I had just caught myself making.

The measurement error and the reading error turn out to be the same error wearing different clothes: wanting a small number of days to settle a question they cannot settle.

So I am not calling it. There is a date in my notes to read the same series over eleven days instead of five, and I will post what it says.

The rule now sitting in the code: a partial day does not look like missing data. It looks like a data point, and you will believe it.

If anything you rely on has a "today" column, go and find out what time today finishes.

devdigest is one tech email a day from more than a hundred sources, no ads, nothing sold on. 14 days free, no card: https://devdigest.io/?ref=indiehackers

I have a filter that strips podcast roundups out of one of my sources. Last week it nearly made me delete the source.

devdigest, the daily tech digest I build and run, pulls from more than 100 sources. New ones go into quarantine: collected and scored normally, but never allowed into anyone's real digest until I've seen a week of actual data and made a call.

Review day came for a batch of 28. One of them, a well-known security writer, read like this in my own tooling:

Stories published in the review window: 0

Stories in the last 30 days: 0

Most recent post: a month ago

Dead feed. Obvious reject. I nearly deleted the entry on the spot.

Then, out of habit rather than suspicion, I opened the actual RSS feed first.

He had published that morning. And weekly before that, with no gaps, going back months.

Here is what happened. When I added the source I also added a title filter, because roughly seven of every eight of his posts are "Weekly Update NNN" podcast roundups rather than articles, and a roundup is not something a digest can use. That filter runs at collection time. Anything it excludes never gets written to disk at all.

So my review tool was never reading what the source published. It was reading what survived my own filter, and displaying it in a column labelled "published in the last 30 days".

The number wasn't wrong exactly. It answered a different question than its label implied, and I had been reading it for a week without noticing.

The general version, which is why this is worth writing down: if you filter data on the way in, every metric downstream is measuring survivors. That is fine while you remember it. It stops being fine the moment someone uses that metric to make a delete decision, because at that point "filtered out" and "never existed" look exactly the same.

The fix was not in the filter. The filter is correct and I kept it. The fix went into the thing that reads the data: any source carrying a filter now says so, in the tool, next to the numbers, with an instruction to check the live feed before calling anything dead.

The source went back into quarantine with a proper window, to be judged on the right question. Not "does he publish", because he clearly does. The real question is whether what remains after the roundups are stripped out earns a permanent slot.

Four days later it caught me again, and that is the part I did not expect. I was about to apply the same kind of fix to a second source, on a number that read "1 story affected". Same trap: that 1 came from records saved before I added that source's filter, so it described a version of the source that no longer exists. The warning I had just shipped is the only reason I checked. Against what the source can actually publish today the real number was 0, and the change would have done nothing at all.

If you run a pipeline that drops records before storage: go and find out what your dashboards have actually been counting.

If you want to see what it actually produces: https://devdigest.io/?ref=indiehackers

Before devdigest, the daily tech digest I build and run, could launch, I had to check something unglamorous: did any of the 100+ sources I pull from explicitly ban what I was about to do?

I expected the answer to track company size. Bigger, more corporate sites, I assumed, would be relaxed about this - they run their own RSS feeds, press APIs, developer-relations teams. Smaller, scrappier publications felt like the ones who'd actually care.

I read the real terms of service for every source, not the privacy policy, and quoted the actual clauses rather than assuming from "all rights reserved" boilerplate (which is close to universal and, alone, isn't real signal - it's the same language that would technically ban what every RSS reader does).

I had it backwards.

A VC-data company's terms ban using content "to train models (including generative artificial intelligence technologies)." A European startup-focused publication bans "any text or data mining or web scraping... for any purpose, including... training, fine-tuning or validation of AI systems." A major infrastructure company's own blog bans bots from "developing, training, fine-tuning... a machine learning model or artificial intelligence (AI) system." All explicit, all recently added, all naming AI training specifically, not generic anti-plagiarism language.

Meanwhile: an engineering-community site's terms say outright, "We permit the posting of a summary and then a link back." A university's news terms say it "offers RSS feeds for syndication purposes." A major code-hosting platform's terms explicitly carve public repositories out of any restriction at all.

The pattern held across the full source list: the sites closest to VC/startup data and big corporate infrastructure wrote the most specific, most recent anti-AI-training clauses. The engineering blogs and syndication-native sites, the ones you'd expect to be precious about their content, mostly weren't.

My best guess why: the well-lawyered sites have their own data-licensing business to protect, so they wrote the clause. The smaller publications' whole distribution model already runs on RSS and being linked to - being read is the product, not the risk.

devdigest only ever excerpts and links back, never reproduces full articles. Four sources still got removed outright once their terms explicitly named this exact pattern. Better to lose a source than build a paid product on top of an explicit no.

If you're building anything that reads other people's content automatically: check the real terms, not just the ones you'd guess would be strict.

https://devdigest.io/?ref=indiehackers

Posted here about devdigest's launch a week ago - quick follow-up on something I found in my own product since then.

I noticed devdigest felt repetitive. It's my own product, so the easy move was to explain the feeling away. Instead I pulled the last 10 issues and counted.

Three sources had appeared in 10 out of 10 days. Between them: 26% of every story I'd been sent - out of a pool of ~1,300 collected daily across 90+ feeds.

The cause wasn't what I expected. devdigest gives every topic you subscribe to a guaranteed slot, so nothing you care about gets crowded out. On any given day, some categories are thin. And in a thin category, a broad high-volume source isn't the best story, it's the only story. One source was winning its guaranteed slot 84% of the time. Not by being good. By being present.

The feature I built to guarantee variety was manufacturing repetition.

The fix was small: any single source can now fill at most one guaranteed slot per person per day. Shipped this week: https://devdigest.io/?ref=indiehackers

What I don't know yet is whether it actually worked. It needs several real sends before the numbers mean anything, so I'm not calling it fixed. Just changed, and watching. Will follow up once there's real data.

The part I'm keeping from this: the fastest way to check a vague feeling about your own product is to go count something.

Hey IH - my first post here.

Full-time network infra engineer, spent the last few months building devdigest.io in my spare time: a daily email curating AI/tech news from 90+ sources into one personalized digest.

Two things I haven't seen elsewhere: every story has a one-line "why you're seeing this" (topic match, a source you've upvoted, or editorial ranking - no black box), and downvoting a story asks why, then actually reweights future picks.

Current state: a handful of subscribers, mostly friends who signed up as a favor rather than real usage - so I genuinely don't know yet if this holds up with strangers. That's the real reason I'm posting here.

Stack: Python/Flask, MySQL, GPT-4o-mini for ranking + tier-aware rewriting, self-hosted on a small VPS. €50/year after a free 14-day trial.

Live sample: devdigest.io/sample - try it: https://devdigest.io/?ref=indiehackers

Appreciate any feedback, especially the critical kind.