Lost users this month and I think I know why — updating in public
Quick honest update on Divide (Chrome extension that
tells you which credit card to use at checkout).
The good: detection is now solid. Spent weeks fixing
cart-total detection across dozens of sites — tips
not updating totals on food delivery, subtotal vs.
total confusion, sites hiding totals in iframes.
That foundation is finally stable. Also got approved
and live on the Chrome Web Store (after getting
rejected first for "keyword spam" — turns out you
can't list retailer names in your description).
The not-so-good: my active users dipped from ~30 to
~26 over the past couple months.
Small numbers, so it's partly noise. But I think the
real reason is the UI. My beta testers told me
repeatedly it showed "too much information" — and I
haven't shipped the redesign that fixes that yet. So
new users hit a cluttered first experience before the
product's intelligence gets a chance to earn their
trust.
The fix is already in progress — my designer is
reworking the whole interface around one idea: show
the decision first, hide the data. Like Google Maps
telling you "Turn left" instead of showing you the
routing algorithm.
Building this solo, bootstrapped, around a job and
school. Slow, but moving.
Question for anyone who's been here: how did you tell
the difference between "normal early churn" and "the
product actually has a problem"? Still learning to
read these small numbers.
I wouldn’t read the ~30 → ~26 drop as proof the product has a real problem yet. Small sample, and probably noisy.The stronger signal is the repeated “too much information” feedback.Feels like the real breakpoint may be: can a new user get to one simple, trusted recommendation fast enough?So instead of asking “is this normal churn or product failure,” I’d split the path:setup → recommendation appears → user understands why → user trusts it → keeps the extension.Your “show the decision first, hide the data” direction sounds right. I’d test it with 5 new/inactive users and watch where they get stuck.
This reframe is genuinely useful — thank you. Splitting
it into setup → recommendation appears → understands
why → trusts it → keeps it turns a vague churn number
into five specific places to look. Right now I don't
know which step is leaking, and that's the actual
problem.
My gut based on beta feedback says the leak is between
"recommendation appears" and "understands why" —
testers trusted the logic when they got it, but several
said they had to work to figure out what they were
looking at. That's the step the redesign targets.
Going to run the 5-user test you suggested once the
redesign ships so I can compare against the old flow.
Appreciate you taking the time on this.
That makes the 5-user comparison clean. With 17 beta testers, 89% saying they’d keep Divide installed, and 100% trust once the recommendation made sense, the redesign only needs to answer one question: how many of the 5 can explain the recommendation without prompting?
thank you, That's the exact test — I'm stealing it. "How many of
the 5 can explain the recommendation without prompting"
is clean because it's binary and I can't fudge it.
Either they get it unprompted or they don't.
And you're right that it isolates the one variable that
matters. Trust isn't the thing to fix — I already have
it once comprehension happens. So the redesign has
exactly one job: make the "why" self-evident before I
say a word. If most of the 5 can explain it back to me
cold, it worked. If they can't, I know the design still
hasn't closed the gap.
Going to run it exactly this way once it ships.
Appreciate you thinking it through with me.
That locks the test. Since “most” technically starts at 3/5, fixing the bar before the redesign ships avoids moving it after the results: would 3/5 count as a win, or does it need 4/5?
Good catch — setting it before results so I can't move
the goalposts after. 4/5.
3/5 feels too soft. If 40% of new users still can't
explain the recommendation, I haven't fixed the
comprehension problem — that's the whole product. 4/5
lines up better with the trust and retention numbers I
already have (100% trusted once they understood, 89%
would keep it). So 4/5 unprompted = clean win, move
toward launch.
The one thing I'll hold loosely: with n=5, the gap
Between 3 and 4 is one person, so if I land exactly at
3 I'm treating it as "close, find what confused the 2"
Rather than a hard fail. But the bar to call it a win
And push forward is 4.
Appreciate you keeping me honest on this. Thank you very much.
4/5 feels clean.For me, a pass would mean they can say both what the recommendation is and why they got it, unprompted.Otherwise they may remember the answer without understanding the reasoning.Are you scoring those separately?
Good catch — and no, I hadn't split them, but you're
right that I need to. Remembering "use the Chase card"
is just recall. Understanding is "use the Chase card
because I earn more on dining and my balance there is
low." Only the second one proves the redesign actually
made the reasoning legible.
So I'm scoring two things per user now:
A pass requires BOTH. Stating the card alone doesn't
count. So the bar becomes: 4 of 5 give both the what
AND the why, unprompted.
That's a stronger and more honest test than what I had.
Appreciate you catching it before I ran it and fooled
myself with recall dressed up as comprehension. thanks
Yep — that closes the loophole. Recall and comprehension are separate now, and requiring both is the honest bar.The only thing I’d freeze before each session is Divide’s expected “why,” so the scoring can’t move after hearing the user’s answer.Once you run the first five, could you post back just three numbers: what / why / both? Even something like 5 / 2 / 2 would make the remaining gap obvious.
Both good additions — taking both.
Freezing the expected "why" before each session is the
missing piece. I'll write down Divide's actual reasoning
for that specific recommendation before the user speaks,
then score their answer against that, not against
whatever sounds close after the fact. Same logic as
locking the 4/5 bar — no drift in the judging either.
And yes, I'll post back the three numbers: what / why /
both. That split is more useful than a single score
anyway — if it comes back like 5/2/2, that tells me
the answer is visible, but the reasoning isn't landing,
which is a totally different fix than if they can't
even find the recommendation.
Genuinely — you've sharpened this test more than a paid
consultant would have. Locking the bar, splitting recall
from comprehension, freezing the expected reasoning, and
now the three-number report. I'll report back as well Thank you for thinking
it through with me really appreciate it.