Diagram playground

archlab 1.0 timeline

title "The checkout outage"

description "What happened on 3 March, in the order it happened"

 

@timeline

period "Before anyone noticed"

event "A routine index migration started on the orders table"

desc "Reviewed, approved, and run at the quietest hour of the week. Nothing about it was unusual."

event "Write latency began climbing, slowly enough that no alert fired"

desc "The alert threshold was an absolute number rather than a rate of change, which is why forty minutes of steady degradation looked like nothing at all."

period "The first hour"

event "Checkout error rate crossed the page threshold"

event "The on-call engineer rolled back the most recent deploy, which was unrelated, and then rolled back the two before it when the error rate did not move"

desc "The migration had run outside the deploy pipeline, so the timeline everyone was reading did not contain the thing that mattered."

event "A second engineer joined and found the running migration"

period "The fix"

event "The migration was cancelled and the table recovered on its own"

event "Checkout drained its queue over the following twenty minutes" #recovery

period "Afterwards"

event "Long-running schema migrations were moved onto the same deploy timeline as everything else, so the one view everybody opens during an incident contains every change"

desc "Nobody argued for it before the outage. The argument was that migrations are not deploys, which was true and beside the point."

event "Latency alerting gained a rate-of-change rule"

event "This document"

desc "Written the same week, while people still disagreed about the order, which is the only time a timeline is worth writing."