The Webhook Notary

The incident

The payment webhook endpoint did what the quick-start guide showed: parse the JSON, mark the order paid. Two problems arrived in the same quarter. First, the provider retried a delivery and the same payment processed twice. Then a security researcher demonstrated the bigger one: the endpoint accepted his hand-crafted 'payment succeeded' event too, because nothing checked who was sending it.

Why the obvious approach fails

A webhook endpoint is a door on the public internet that changes your system's state — and quick-start code treats it like a trusted internal call. It isn't. Anyone who finds the URL can POST to it, so unverified payloads are an open invitation to forge the exact events you care most about. And providers retry by design — networks drop acknowledgments — so an endpoint without deduplication will process real events twice. Inbound events need the same rigor as payments themselves: prove the sender, process each event once, and keep the ones you couldn't handle.

The standard

• +Every inbound webhook verifies the provider's signature before parsing; unsigned or invalid payloads are rejected and logged as security events.
• +Event IDs are deduplicated: a retried delivery is acknowledged and ignored, never reprocessed.
• +Handlers acknowledge fast and process async (through the Job Marshall) — the provider's timeout never dictates your processing time.
• +Failed processing lands in a dead-letter queue with the full payload; events are replayable after the bug is fixed.
• +Out-of-order arrival is handled by design — event timestamps and state checks, not the assumption that the network preserves sequence.

Composes with