Runbook template: on-call runbooks that work at 2 a.m.

The page goes off at 2 a.m. You open the alert, click the runbook link, and land on a wiki page last edited two years ago by someone who has since left. It describes a dashboard that no longer exists and ends with "escalate if needed." So you do what everyone does: open the code, guess, and wake up the one person who knows.

A good runbook turns that into ten calm minutes. This guide gives you a template, a filled example for one alert, the config to put a runbook link on every page, and the habits that stop runbooks from rotting.

Why runbooks are worth the effort

The Google SRE book puts it plainly: recording best practices ahead of time in a "playbook" produces "roughly a 3x improvement in MTTR as compared to the strategy of 'winging it.'" It also says no playbook replaces engineers who can think on the fly. The point is to take the routine thinking off the table so the on-call person can spend their attention on what is actually new.

The SRE workbook's on-call chapter describes what these documents contain: the severity and impact of the alert, debugging suggestions, and possible actions to mitigate and fully resolve it. It adds that whenever an alert is created, a matching playbook entry is usually created too.

Runbooks vs playbooks

The words are used inconsistently, so it helps to pick definitions and stick to them.

• Google SRE calls the per-alert response guide a "playbook" (see the quotes above).
• PagerDuty calls a step-by-step guide for a specific task or issue a runbook, and describes a playbook as the broader response to larger events, which "can include multiple runbooks and team members."