Root cause analysis

Root cause analysis is the structured investigation into why an incident happened, conducted after service has been restored. The distinction that makes it worth doing is between the trigger and the conditions: the deploy that broke production is the trigger, and the absence of a canary stage, the missing test, and the alert that fired too late are the conditions that let a routine deploy become an outage.

The singular in the name is the problem with it. Serious incidents in distributed systems almost never have one cause; they have a set of contributing factors that were individually survivable. Insisting on a single root cause tends to produce the last human action in the chain, which is both the least useful answer and the one most likely to make people cautious about reporting things. Practitioners increasingly talk about contributing factors instead, for exactly this reason.

The technique most associated with it, asking why repeatedly, works if it is pointed at the system and stops being useful the moment the answers become about a person. "Why did the deploy break production" leading to "because nothing tested that path" is progress; leading to "because Sam did not check" is a dead end, because the next incident will involve a different person and the same missing test.