How to Run an Incident Postmortem

The purpose is not to establish what went wrong. It is to change something so it cannot go wrong the same way again.

How to Run an Incident Postmortem — Troiana insight cover

In short

A postmortem is a structured review after an incident, focused on the conditions that allowed it rather than on who acted. It works when it is blameless — because people describe what actually happened only when it is safe to — and when it ends in a small number of owned, dated actions. A postmortem producing no changes was a meeting.

What it is for

One thing: make this class of failure less likely or less damaging next time.

Not assigning responsibility. Not producing a document for whoever asks. If the review ends without something concrete changing, it was theatre.

That framing decides everything else, including why blamelessness is practical rather than merely kind.

Why blameless is not softness

People describe what actually happened only when describing it is safe.

The moment a review can end in someone being blamed, accounts become careful. The engineer who ran the command does not mention that the confirmation prompt looked identical to a harmless one. That detail — the one that would prevent recurrence — is exactly what you lose.

So the operating assumption is: everyone acted reasonably given what they knew at the time. If someone made the wrong call, the question is what made the wrong call look right. That is a fixable condition. "They should have been more careful" is not.

The document

Keep it short enough to be read.

A summary. What broke, for whom, for how long, in three sentences. Most readers will read only this.

Impact, in user terms. Not "the queue backed up" but "1,400 customers could not complete checkout for 38 minutes". Impact in system language understates and is easy to dismiss.

A timeline. When it began, when it was detected, when someone was working on it, when it was mitigated, when it was resolved. Timestamps matter more than prose here.

What happened. The chain of conditions, in plain language.

What went well. Genuinely — a fast rollback, a useful alert, a good handover. This is not decoration; it identifies the practices worth keeping.

Actions. Each with an owner and a date.

The gaps that matter more than the cause

The timeline usually reveals more than the root cause does. Three intervals are worth measuring every time.

Start to detection. If it took 40 minutes to notice, that is a monitoring gap — and it will apply to the next incident too, whatever caused this one. Often the highest-value finding.

Detection to response. Was the alert seen? Did it reach someone who could act?

Response to mitigation. Did they know what to do, or spend 20 minutes finding out?

Fixing detection time improves every future incident. Fixing one specific cause improves one.

Getting past the first answer

The first explanation is usually a symptom.

The site went down. Why? The database ran out of connections. Why? A background job opened connections without closing them. Why? An error path skipped the cleanup. Why did nobody notice? There was no alert on connection pool saturation, and the job's errors were logged but not tracked.

The first answer suggests restarting the database. The last suggests two changes that would have caught it — and would catch the next one.

Keep asking why until you reach something you can actually change. Stop when the next answer is a fact about the world rather than a decision.

'Human error' is never a root cause

When an account ends at "someone made a mistake", the review stopped early.

People make mistakes constantly; that is a constant, not an explanation. The useful questions are why the mistake was possible, why it was not caught, and why the consequence was severe.

A command that deletes production data with no confirmation, no dry run, and no backup is a system that will eventually be operated by a tired person on a Friday. The fix is in the system, not in the person.

Actions that actually get done

Most postmortem actions are never completed, for predictable reasons.

Keep them few. Three real changes beat twelve aspirations. Pick the ones that would have prevented or shortened this incident.

Give each an owner and a date. "The team will improve monitoring" belongs to nobody.

Put them where work lives. In the backlog, prioritised against everything else, not in a document nobody reopens.

Prefer prevention, then detection, then response — in that order of value, but detection improvements are often cheaper and help more broadly.

Review them at the next postmortem. Nothing kills follow-through faster than actions that quietly expire.

Which incidents deserve one

Anything that affected users meaningfully. Anything where you got lucky — a near miss is a free lesson, and reviewing those is the cheapest learning available. Anything that took surprisingly long to diagnose, even if impact was small, because diagnosis time is a recurring cost. And anything that recurred, which means the previous fix did not work.

Not every glitch. A postmortem process applied to everything becomes paperwork, and paperwork gets ignored.

The measure of a good process

Not the quality of the documents. Whether incidents get shorter.

If detection time is falling, if the same class of failure stops recurring, if people volunteer information without being asked — the process is working. If you have a folder of well-written postmortems and the same outage every quarter, it is not.

If you are setting up an on-call process and want the review side right from the start, book a call.

Common questions

What is a blameless postmortem?

A review that examines the conditions allowing an incident rather than who acted, on the assumption everyone behaved reasonably given what they knew. It is practical rather than merely kind: people only describe what actually happened when it is safe to do so, and the details they withhold under blame are exactly the ones that prevent recurrence.

What should an incident postmortem include?

A three-sentence summary, impact stated in user terms rather than system terms, a timestamped timeline, the chain of conditions in plain language, what went well, and a small number of actions each with an owner and a date. Short enough that people actually read it.

Why is 'human error' not a root cause?

Because people make mistakes constantly — that is a constant, not an explanation. The useful questions are why the mistake was possible, why nothing caught it, and why the consequence was severe. A destructive command with no confirmation, dry run, or backup will eventually be run by a tired person; the fix belongs in the system.

Which part of the timeline matters most?

Usually the time between the incident starting and someone detecting it. A monitoring gap applies to every future incident regardless of cause, so shortening detection improves all of them — whereas fixing one specific cause improves one.

Why don't postmortem actions get completed?

Because there are too many, they have no named owner or date, and they live in a document rather than in the backlog where work is actually prioritised. Three owned, dated changes beat twelve aspirations, and reviewing outstanding actions at the next postmortem is what stops them quietly expiring.

Have something worth building right?