How we took a bolt-built storefront from one Black Friday to the next — without dropping an order.

A storefront that worked — until it got busy.

The storefront had shipped, gotten traction, and made real revenue. Then last Black Friday the queue started silently dropping messages under load, and a payment-webhook race meant some orders processed twice while others didn't process at all. Nobody knew in real time because there were no alerts.

The next Black Friday was eleven weeks out. The task was to make the order path reliable under 3× peak before that date — not as a side project, but as the only project.

Five problems that made peak events dangerous.

We read every order flow and load-tested at 2× last year's peak before changing anything.

1. 01The order queue dropped messages silently when memory pressure exceeded the in-process limit.Critical
2. 02Checkout was not idempotent — a duplicate webhook created a duplicate order with no deduplication key.Critical
3. 03There was no explicit order state machine — orders could reach contradictory states concurrently.High
4. 04Inventory updates were not atomic, producing race conditions under concurrent checkout.High
5. 05No alerting on order failures — the team learned about problems by refreshing the orders page.High

Fix the order path, prove it under load.

We worked from highest risk down. The queue and the checkout path came first because they were the failure modes that lost money. Everything else could wait. And nothing shipped until it had passed a load test at 3× expected Black Friday peak against a production data copy.

Every fix was built to be rollback-safe and dark-deployable. There was no scope for a rollout that couldn't be undone in ten minutes.

Six fixes, in the order they shipped.

1. Made checkout idempotent.

   Added idempotency keys on the payment webhook and an upsert-based checkout path. A duplicate webhook now produces the same order exactly once.

2. Moved to a durable external queue.

   Replaced the in-process queue with SQS. Messages are durable, dead-lettered on failure, and visible — the queue can't silently lose work under load.

3. Built an explicit order state machine.

   Orders move through explicit transitions enforced in the database with optimistic locking. Concurrent updates compete cleanly instead of producing contradictory states.

4. Fixed the inventory race.

   Inventory decrements moved to atomic database operations with a pessimistic lock on the product row during checkout. Overselling became structurally impossible.

5. Replaced page-refreshing with alerts.

   Sentry alerts on order failures within 60 seconds. Dead-letter queue depths and checkout error rates are on a Grafana dashboard the team has open during peak.

6. Load-tested at 3× expected peak.

   k6 test against a production data copy at 3× last year's Black Friday. Zero dropped orders, p95 checkout under 400 ms. The test runs in CI before every deploy.

A storefront you can watch on a dashboard.

Black Friday came. 312% more orders per hour than the prior year. Zero dropped checkouts. The team watched dashboards instead of refreshing the orders page. The DLQ never grew. The load test had told us exactly what to expect.