Designing webhook consumers that survive retries

If your handler assumes each event arrives exactly once, it will eventually double-charge someone, or double-issue a pass. Retries are not a failure mode to be avoided; they are the contract.

Key on the event, not the payload

Every delivery carries a stable event id. Record it before you act, and treat a repeat id as a no-op. Deduplicating on payload contents breaks the moment a field is added.

Do not assume order

registration.updated can arrive before registration.created. Handlers that require ordering should reconcile against current state rather than replaying a sequence — fetch the record, apply what is true now.

Accept events you do not know

New event types get added. A consumer that errors on an unrecognised type turns a routine addition into an outage. Log it, acknowledge it, move on.

Acknowledge fast, work later

The delivery timeout is not the place to run your business logic. Return 2xx once the event is durably queued, then process it on your own schedule. Anything slower turns a downstream slowdown into a retry storm.