How to Write a Runbook That Actually Gets Used in an Incident
It is 2:15 AM. A PagerDuty alert fires for a database CPU spike. The responder, who has not touched this service in six months, opens the runbook linked in the alert.
They find a 14-page Confluence document. It starts with the history of a migration project from three years ago. The first command is buried on page four. The responder closes the tab and searches Slack, because of course they do.
Most runbooks are write-once artifacts that become stale the moment an incident happens differently than expected. They are too long, too generic, or written for people who already know the system. Under stress, engineers skip them entirely. Google SRE puts documented escalation paths and response tactics in the preparation work, before an incident begins.
A runbook that gets opened starts with a decision, not a history lesson. If rds-primary-cpu-95-5m fires, run the CPU check below and expect five values above 95.0. If the result differs, mark the alert recovered. If the command times out, page @database-oncall in #inc-payments. That is the whole job.
The Anatomy of a Panic-Proof Document
The first 200 words should function as a decision tree. Context and architecture explanations belong below the operating path, where they can help the person who needs them without slowing everyone else down.
Here is what a first screen can look like. The service names are deliberately concrete. In a real runbook, use the real names, not a blank for someone to solve at 2:15 AM.
Expect: the five most recent Average values to be above 95.0.
If they are: page @database-oncall in #inc-payments, then continue to the lock-review branch. If they are not: mark the alert recovered and attach the command output to the incident record. This check normally takes 30 seconds. If the command has not returned after two minutes, proceed to rollback or escalate to #platform-oncall.
That is more useful than a generic note saying “investigate database performance.” AWS says a runbook should contain the minimum information needed to achieve a specific outcome, including expected outcomes, escalation timeframes, reversible actions, and success verification. Microsoft’s playbooks use prerequisites and a logical workflow before deeper investigation steps.
Use literal, copy-paste-ready commands. State the expected output after each one. Add the clock where it matters. A responder should never have to infer whether a command succeeded, how long to wait, or what to do when it does not.
Treating Words Like Code
A runbook should map to one alert or one failure symptom, not an entire service. “Database is slow” is a topic. “Primary RDS instance CPU at 95% for five minutes” is an entry point.
If “depends on the scenario” appears more than once, split the page. One runbook can own a primary CPU saturation alert. Another can own connection exhaustion. A third can own replica lag. Google measures alert quality by precision, recall, detection time, and reset time, all of which argue for an alert that names a specific event instead of a vague condition.
Keep the files in version control beside the services they document. A pull request that changes deployment steps, adds a dependency, or changes an alert threshold should include the matching runbook update. AWS’s public playbook samples show how a standard template must be adapted to a team’s risks, tools, and working practices.
Then give the file a visible operating history. After every incident, add a Last Tested timestamp and the name of the person who ran it. Archive anything that has gone unvalidated for six months. A dead runbook does more than waste storage. It teaches the team that the next page may be fiction.
That maintenance burden is the actual problem. NIST recommends incorporating incident response throughout risk management to improve preparedness, efficiency, and recovery, rather than reserving it for the outage itself. Strong runbooks come from inserting updates into deployment and incident workflows, instead of treating creation as a one-time writing project.
The Person Who Did Not Write It
The person running your runbook at 2 AM is often not the person who built the service. Assume they have production access and enough context to follow precise instructions, but no private map of every nickname, dependency, or past incident.
Write in the vocabulary that appears in dashboards, alerts, and commands. Link internal terms to a glossary when they need a definition. Name the owner and escalation channel. AWS’s guidance requires clear ownership and expected outcomes, along with contact information, expected notifications, and response steps.
Make blast radius explicit before a risky action. “This command restarts all customer-facing API nodes. Expect 30 seconds of 503s.” That sentence lets the incident commander decide deliberately, instead of discovering the impact when customers do.
Test the page in an exercise where the author is absent, Slack is off limits, and the only assistance is the runbook. If the responder has to search the web or ask a basic question, record the missing step and update the document. Rotate who runs the drill. If it only works when the principal engineer runs it, you have documented tribal knowledge instead of a dependable operating procedure.
CISA’s tabletop exercises use scenario-based practice and after-action reporting to update information-sharing and recovery procedures. Google SRE also calls for hands-on exercises and periodic review of response procedures.
When to Throw It Away
Finally, make the right page easy to find. Link it directly from the alert description. Use a naming convention that mirrors the alert name, such as rds-primary-cpu-95-5m-runbook.md, and pin the runbook index in the incident-response channel. If people cannot find it in under 10 seconds during an outage, they will not use it. CISA says procedures should help teams identify, coordinate, remediate, recover, and track successful mitigations.
Lean teams have less room for operational process that lives in people’s heads. That makes an up-to-date runbook more valuable, not less. Tools that generate structured documentation from deployment pipelines, tickets, and code commits can give a small team a stronger starting point. Doc Holiday generates runbooks and incident-response documentation from engineering workflows, then keeps that foundation synchronized as systems change so teams can validate and refine it instead of rebuilding from scratch every time.
