Skip to content
arch-lab

The checkout outage

What happened on 3 March, in the order it happened

Open in the playground
BEFORE ANYONE NOTICEDTHE FIRST HOURTHE FIXAFTERWARDSA routine index migration started on the orders tableReviewed, approved, and run at the quietest hour of the week. Nothing about it was unusual.Write latency began climbing, slowly enough that no alert firedThe alert threshold was an absolute number rather than a rate of change, which is why fortyminutes of steady degradation looked like nothing at all.Checkout error rate crossed the page thresholdThe on-call engineer rolled back the most recent deploy, which wasunrelated, and then rolled back the two before it when the error rate didnot moveThe migration had run outside the deploy pipeline, so the timeline everyone was reading didnot contain the thing that mattered.A second engineer joined and found the running migrationThe migration was cancelled and the table recovered on its ownCheckout drained its queue over the following twenty minutesLong-running schema migrations were moved onto the same deploy timeline aseverything else, so the one view everybody opens during an incident containsevery changeNobody argued for it before the outage. The argument was that migrations are not deploys,which was true and beside the point.Latency alerting gained a rate-of-change ruleThis documentWritten the same week, while people still disagreed about the order, which is the only timea timeline is worth writing.