My friend in med school kept complaining about NotebookLM, so I built the fix

Ha, the "am I just feeding the next training set" feeling is real. And yeah, having to bring all your own sources is exactly the friction — NotebookLM is basically a reader you have to feed, not something that goes and finds stuff for you. The "point it at my project folder and let it work" idea is a whole different (and honestly harder) product, but I get why you'd want it.

And hey — go post your project. Seriously. I sat on this way too long telling myself it wasn't ready, and the second I posted it I learned more in a day than in the weeks of polishing before. Ship the messy version. Nobody's judging as hard as you are.

You put it better than I did in the actual post. "The point where NotebookLM stops being useful" — that's exactly the line I've been circling around and couldn't name.

That gap between generating knowledge and using it is the whole bet. Adding more AI is a race I'd lose against Google anyway; owning the workflow after the AI is done feels like the only defensible spot for a small solo thing. The cross-notebook search especially — that one came straight from watching someone drown in a dozen separate notebooks with no way to look across them.

Really appreciate you taking the time to think it through like this.

Ha, we're in the exact same boat — riding undocumented internals someone else can yank out from under us. Solidarity.

On the early-warning question: yeah, I just wired this up — funnily enough another commenter on this same post nudged me toward exactly this ("wire an 'it broke' signal so you hear it from telemetry before angry users"), and it finally got me to stop putting it off. It's dead simple. Every selector that matters has a stable id, and when a scrape comes back with zero matches — zero, not "fewer than expected," just nothing — the extension fires a tiny ping to my worker: site, which selector, extension version. No user content, nothing sensitive, just "this hook returned empty." I throttle it to once a day per user per selector so one bad redesign doesn't hammer me with thousands of identical pings.

The nice part is the signal is basically unambiguous. A selector that used to return 20 nodes and now returns 0 isn't a fluke — it means the markup moved. So instead of a user emailing me "your thing is broken" the night before an exam, I see it on a dashboard the morning it ships and can go fix the one selector.

The thing I haven't solved: partial breakage. Zero matches is loud and obvious. But if a redesign makes my selector match the wrong element and I quietly pull garbage, that sails right past the zero-check. Still thinking about how to catch that without a human eyeballing output. If you crack that one for your OS-internals case I'd genuinely want to hear it.

And agreed on the citation-on-the-back thing — that's the one detail I'm not letting anyone talk me out of.

Honestly? At this stage I can't tell from the numbers, and I think pretending I can would be the mistake. With a handful of users, any "drop-off went down" is just noise — I'd be reading tea leaves and calling it a metric.

So early on I'm not really trusting quantitative signals at all. What I'm doing instead:

• Watching, not measuring. I'd rather sit on a call and watch one med student actually try to get their flashcards into Anki than stare at a funnel. If they stop talking, squint, or go "wait, how do I…", that's the signal. You can't fake that and you don't need 100 people to see it.
• Fixing only things I understood the "why" of. If someone bounces and I don't know why, I don't touch anything — a blind tweak on a mystery is how you convince yourself noise is a trend. I'd rather ask them than guess.
• Treating early changes as "did this remove a specific, named frustration for this specific person," not "did this move a number." The number can wait until there's enough traffic for it to mean something.

Basically I think the trap you're pointing at is real: the temptation to A/B-think your way through a stage where you don't have the sample size to A/B anything. Early on the honest instrument is a conversation, not a dashboard.

Curious how you've drawn that line yourself — at what point did numbers actually start telling you something true, versus just feeling like they should?

You just listed my roadmap back to me, which is either encouraging or slightly spooky. Quick status on all three, since I've actually poked at each:

Page-numbered citations — this is the one I most wanted and the one that fought back hardest. I spent an evening in the network tab watching what NotebookLM actually returns, and the bad news is it only hands back character offsets into the source text, never page numbers. The UI itself doesn't have them. So the only path is grabbing the original PDF and computing the page myself with pdf.js — doable in theory, but it hinges on getting the raw file, which isn't guaranteed. I've got it written down as the hardest, highest-value thing on the list.

Cross-notebook search — this one already ships, but with an honest asterisk: it's keyword, not embeddings. Semantic search across notebooks is the obvious next rung and I know it, I just haven't wanted to promise vectors before the plain version is actually good.

Figure/table extraction — that one's new to me and it's a good idea. Adding it to the list.

The through-line on all of these is the same wall: no public API, so I'm reading whatever the page and its private endpoints will give me. Which is exactly why "even in edge cases they won't notice" is the whole game. Genuinely appreciate the specifics — vague "add more features" comments I can't do anything with; this I can.

This is one of the most useful comments I've gotten. Thank you for actually thinking it through.

You're right that the Anki gap is the real product. I framed the post around citations because that's what I personally feel, but the flashcards-that-can't-reach-Anki thing is the one that costs a real person real hours every week — and that's a much sharper reason to exist. I'm doing a clean .apkg export; AnkiConnect for people who have the desktop app open is a great call, hadn't thought about detecting that. Adding it to the list.

The page-number point is the one I'm going to sit with. You nailed exactly why it's hard: NotebookLM hands you a chunk, not a page, so you're reverse-mapping the chunk back onto the source PDF yourself. It's annoying enough that everyone else punts and outputs "source 3, somewhere in a 40-page PDF." If I can get reliable page-level refs, that's the thing worth paying for. That's now the hard problem I actually want to solve.

And the DOM warning — I laughed, because I literally spent all of yesterday on exactly this. Google had quietly changed a selector, my code was matching zero elements, and the tool was "working" while silently saving half of every conversation. I ended up writing a little probe script just to tell me the truth about the live page. A telemetry "it broke" signal is the obvious next step — hearing it from a dashboard instead of an angry student the night before an exam. Putting that in.

Seriously, thanks. You just reordered my roadmap.