Black Friday, One Missing Index, and 53 Minutes of Checkout Pain

I was the primary on-call engineer for a mid-size e-commerce company doing roughly 50k orders a day outside of peak season. Checkout lived in a Node.js monolith on ECS with PostgreSQL behind it, and most of the year that setup was boring in the best possible way. We knew Black Friday would be 5x to 8x a normal day, and we had load-tested the app tier, but we had not replayed the exact mix of promotions marketing planned to run that weekend.

At 6:02 AM, p99 for /api/checkout jumped from a few hundred milliseconds to double digits. Two weeks earlier we had rebuilt the promotions table during a merchandising change and missed a composite index on (product_id, valid_until). On an ordinary Tuesday the query still came back fast enough that nobody noticed. Under Black Friday concurrency, the planner fell back to a sequential scan on a hot table, request threads stacked up behind the database pool, and by 6:15 AM customers were getting spinning checkouts and timeout errors.

We added the missing index with CREATE INDEX CONCURRENTLY, kept promo stacking off until the build finished, and watched checkout recover within minutes. The useful follow-up was less glamorous: schema diff checks in CI, pre-event load tests using production-like discount data, and a separate alert on pool saturation so we do not wait for endpoint latency to become a customer problem before waking someone up.