I Built a Webhook Reliability Company. Then My Own Webhooks Broke

September 4, 2026

Funny thing happened last week. I run Hookpilot, a webhook reliability platform, so my entire pitch to customers is "stop trusting webhook delivery order, treat every event as unreliable, build for that." Then I went and got bitten by the exact thing I warn people about, on my own billing system, in production.

Here's what happened.

A user cancels their subscription. Paddle (our payment processor) defers it to end of billing period, like most processors do. Nothing charges again, access continues until the date, then it lapses. Standard stuff. Except our own reminder email didn't know the difference between "will renew" and "about to end," so a customer who'd already cancelled got an email telling them their subscription renews soon. Not a great look for a company selling reliability.

So I fixed it. Added a field that tracks the pending cancellation, wired it into the webhook handler, built a proper "keep my plan" button, shipped it, tested it, felt good about it.

Then I tried it on my own account and it didn't work. The field stayed empty even though I'd genuinely cancelled.

I could've guessed at a dozen causes. Instead I pulled the actual webhook delivery logs from Paddle's dashboard. Turned out Paddle had fired two subscription.updated events for one cancellation, 185 milliseconds apart. One carried the cancellation details, one didn't. My server processed them, and the one without the cancellation data landed second, wiping out the correct state the first one had just set.

Nothing was wrong with either webhook. Paddle doesn't promise they'll arrive in the order they happened, and mine hadn't. Whichever one my server happened to process last, won, regardless of which one was actually more recent.

That's the exact scenario I built Hookpilot to protect other people's endpoints from. I'd just never applied the lesson to my own backend, because "my own billing system" didn't feel like the kind of place that needed the same paranoia as customer-facing webhook infrastructure. It was.

The fix was small once I knew what to look for: stamp every applied event with its own timestamp, and refuse to let an older one overwrite a newer one. A few lines of code. The part that actually mattered was realizing delivery order isn't something you get to assume anywhere, including in your own house.

If you're running anything that consumes webhooks, and you're not already checking whether the event you're processing is actually the newest thing that happened, this is your reminder. It won't be obvious until it shows up on your billing page, or worse, until something a customer notices before you do.

Hookpilot exists so other people don't have to learn this the hard way. I still did.

Festybee

posted to

Hookpilot

Say something nice to ironman007…

Post Comment

1

Good question, and honestly I think the answer is I dodge it rather than solve it. The 185ms thing was me untangling my own billing state as a Paddle customer — two webhooks landing basically together, had to decide which one wins. But for webhooks Hookpilot delivers to customers, I don't try to guarantee freshness at all. Docs are pretty blunt about it: at-least-once, not ordered. A retry can land after a newer event for the same source, so treat the webhook as "something changed, go check" rather than the actual state. Which I think dodges your question more than answers it — if you carry a freshness check to the point of action, you've just moved the race condition, not closed it, because state can still shift in the gap between your check and the side effect actually firing. The only thing that really closes it is re-fetching from the provider right before you act and treating the webhook as a trigger, not a payload. So yeah, I think customers do need to carry that all the way to the action, and I don't think a webhook layer can do it for them — only the app doing the consuming knows what "still authoritative" even means for that action. All I can do is not pretend the event stream is more trustworthy than it is, so nobody builds an auth check on an ordering guarantee that was never real. How's OpsWatch handling it — re-check baked into the workflow engine, or left as a contract the workflow author has to honor?

Festybee

·
3 days ago
·
Reply
1. 1
  
  I think that’s exactly the boundary. OpsWatch doesn’t assume the webhook layer can determine authority. Like you said, only the consuming system knows whether the action is still permitted in the current context. The part I’m trying to make explicit is that the check can’t just be “we re-fetched recently.” For a consequential action, the system needs evidence that the authority being relied on was still valid at the point the side effect was actually allowed to occur. So I’m treating freshness as an assurance property, not just a workflow convention. If the consuming app checks state at T0, dispatches at T1, and the relevant authority changes in between, then unless the enforcement boundary rejects that stale authority, OpsWatch would not treat the action as safely covered just because the app performed a re-check. That’s why I’m separating: authorization → dispatch → execution → downstream evidence. Where the system can’t establish the later steps, I’d rather preserve the uncertainty than collapse it into “blocked” or “safe.” Your webhook model actually seems quite clean in that sense: the event says “something changed”; the app decides what that means; and the real assurance question begins at the action boundary.
  
  SuperMcG
  
  ·
  3 days ago
  ·
  Reply
1

The 185ms example is a great illustration of why “latest thing processed” and “latest thing that actually happened” can't safely be treated as the same state. There's another boundary here I'm curious about. Once that state is used to authorize a consequential action, fixing event ordering alone may not be enough. Imagine the cancellation state is correct when a workflow begins, but another billing event changes the entitlement before the eventual side effect. At that point the historical state was valid — it's the authority derived from it that has gone stale. I've been working on this distinction with OpsWatch: separating “this was valid when evaluated” from “this was still authoritative when the consequence occurred.” Does Hookpilot stay deliberately at the event/state-consistency layer, or do you see customers needing to carry that freshness check all the way to the action that consumes the webhook state?

SuperMcG

·
3 days ago
·
Reply
1. 1
  
  Good question, and honestly I think the answer is I dodge it rather than solve it. The 185ms thing was me untangling my own billing state as a Paddle customer — two webhooks landing basically together, had to decide which one wins. But for webhooks Hookpilot delivers to customers, I don't try to guarantee freshness at all. Docs are pretty blunt about it: at-least-once, not ordered. A retry can land after a newer event for the same source, so treat the webhook as "something changed, go check" rather than the actual state. Which I think dodges your question more than answers it — if you carry a freshness check to the point of action, you've just moved the race condition, not closed it, because state can still shift in the gap between your check and the side effect actually firing. The only thing that really closes it is re-fetching from the provider right before you act and treating the webhook as a trigger, not a payload. So yeah, I think customers do need to carry that all the way to the action, and I don't think a webhook layer can do it for them — only the app doing the consuming knows what "still authoritative" even means for that action. All I can do is not pretend the event stream is more trustworthy than it is, so nobody builds an auth check on an ordering guarantee that was never real. How's OpsWatch handling it — re-check baked into the workflow engine, or left as a contract the workflow author has to honor?
  
  Festybee
  
  ·
  3 days ago
  ·
  Reply
1

The production incident makes the problem much more tangible than the usual webhook reliability pitch. The harder commercial question seems to be whether teams buy protection before experiencing this themselves—what have you seen that indicates the pain is strong enough to trigger proactive spend?

Aryan Sinh

·
10 days ago
·
Reply
1. 1
  
  Good question, and honestly the hardest one to answer comfortably.
  Most of what's actually working right now is reactive, not proactive. I write content around the exact symptoms people search for after something's already broken — "webhook returns 200 but nothing happens," "why do I keep getting duplicate webhook events" — and those convert because whoever's typing that is already mid-incident. Nobody's Googling "webhook reliability platform" before they've been burned once.
  What I don't have is a clean story where someone bought protection purely on the strength of the pitch, with zero prior pain. I'm not going to pretend otherwise.
  The one lever I've got against that is the free Solo tier — the bet isn't "convince someone to buy before they've felt pain," it's "get them using it for free before they've felt pain, so the pain just... doesn't happen to them, or it happens on the free tier where the blast radius is small." That's a much lower bar than a purchase decision. Whether it actually holds someone long enough to become a paying customer once they scale past what free covers, I don't have enough volume yet to say with a straight face.
  
  Also, I could compare this to Insurance. Would you likely wait for the incident before buying insurance? No. The difference here is that there is no free-tier insurance.
  
  Festybee
  
  ·
  10 days ago
  ·
  Reply
  1. 1
    
    That reactive-vs-proactive distinction is the interesting commercial question. I’d be interested in digging into whether the free tier actually creates that later willingness to pay as teams scale. Happy to continue privately — what’s the best email to reach you on?
    
    Aryan Sinh
    
    ·
    10 days ago
    ·
    Reply