Incident Management & Post-Mortems

Chapter 42: Incident Management & Post-Mortems

Learning Objectives

Understand the full incident lifecycle from detection to closure
Know the roles in incident management (IC, Comms Lead, SME)
Write effective blameless post-mortems
Apply the “5 Whys” and fishbone analysis
Build runbooks that actually help during an incident

What is this? (Beginner On-Ramp)

Incident Response is the structured process of dealing with a massive production outage. It brings order to chaos by assigning specific roles (like Incident Commander), stopping the bleeding quickly, and then figuring out how to prevent it from ever happening again.

29.1 Incident Lifecycle

    Incident Lifecycle
    ───────────────────

    ┌───────┐     ┌───────────┐     ┌──────────┐     ┌──────────┐
    │DETECT │────►│  RESPOND  │────►│ RESOLVE  │────►│ LEARN    │
    └───────┘     └───────────┘     └──────────┘     └──────────┘
       │               │                  │                │
    Alert fires    Acknowledge        Fix applied       Post-mortem
    User reports   Incident declared  Verify fix        Action items
    PagerDuty      Roles assigned     Communicate       Prevention

    Key Metrics:
    ─────────────
    MTTD: Mean Time to Detect
    MTTA: Mean Time to Acknowledge (< 5 min for P1)
    MTTR: Mean Time to Resolve/Recover
    MTTF: Mean Time to Failure (between incidents)

    Goal: Minimize all MT** values

29.2 Incident Severity Levels

    Severity Classification
    ────────────────────────

    SEV 1 (Critical):
    • Service completely down for all users
    • Data loss or corruption occurring
    • Security breach in progress
    → Response: Immediate, all hands
    → Customer comms: Within 15 minutes
    → Target MTTR: < 1 hour

    SEV 2 (High):
    • Major feature unavailable for many users
    • Significant performance degradation (>50% slower)
    • Partial outage affecting a region/segment
    → Response: Immediate during business hours
    → Target MTTR: < 4 hours

    SEV 3 (Medium):
    • Non-critical feature degraded
    • Performance degradation affecting minority of users
    • Internal tool/dashboard down
    → Response: Same business day
    → Target MTTR: < 24 hours

    SEV 4 (Low):
    • Cosmetic issues
    • Single-user problem
    • Monitoring gap (alert isn't firing but should)
    → Response: Normal ticket queue

29.3 Incident Roles

    Incident Command Structure
    ───────────────────────────

    ┌─────────────────────────────────────────────────────────┐
    │  Incident Commander (IC)                                 │
    │  • Single decision maker                                │
    │  • Delegates tasks                                      │
    │  • Does NOT do technical work (stays high-level)        │
    │  • Controls the incident channel                        │
    └──────────────────────┬──────────────────────────────────┘
                           │
         ┌─────────────────┼──────────────────┐
         ▼                 ▼                  ▼
    ┌──────────┐    ┌──────────────┐    ┌──────────────┐
    │  Comms   │    │   SMEs       │    │   Scribe     │
    │  Lead    │    │ (Subject     │    │              │
    │          │    │  Matter      │    │ Documents    │
    │ Customer │    │  Experts)    │    │ timeline     │
    │ updates  │    │              │    │ and actions  │
    │ Status   │    │ Debug and    │    │              │
    │ page     │    │ fix the      │    │              │
    └──────────┘    │ actual issue │    └──────────────┘
                    └──────────────┘

IC Responsibilities During an Incident

    IC Script (first 15 minutes)
    ──────────────────────────────

    1. Declare the incident (in Slack/PagerDuty):
       "I'm declaring a SEV 2 incident. IC: @alice. Comms: @bob.
        SMEs needed: @backend-team, @database-team"

    2. Create incident channel: #incident-2024-01-15-checkout

    3. Establish shared situation awareness:
       "Current understanding: Checkout is failing for 30% of users.
        Started ~14:35 UTC. First affected report at 14:38."

    4. Assign tasks explicitly:
       "@alice: Check recent deployments"
       "@bob: Update status page - investigating"
       "@carol: Pull error logs from checkout service"
       NOT: "Someone check the logs" (nobody does it)

    5. Timebox investigations:
       "I need an update from each team in 10 minutes"

    6. Keep channel focused:
       "Theory discussions in a thread, key findings here"

29.4 Communication Templates

# Status Page Update Templates

### Initial (within 15 min):
We are investigating reports of [service] issues.
Approximately X% of users may be affected.
We will provide an update in 30 minutes.

### Investigation:
We have identified the likely cause as [component/change].
Our engineering team is actively working on a fix.
Next update in 20 minutes or sooner if resolved.

### Resolved:
The issue affecting [service] has been resolved as of [time].
[Brief cause explanation if appropriate]
We are monitoring the service and will provide a post-mortem.

# Internal Incident Slack Template
## Incident Update — [TIME]
**Status**: Investigating / Mitigating / Resolved
**Impact**: [X]% of [feature] requests affected
**Current Action**: [What team is doing RIGHT NOW]
**Next Update**: In [N] minutes or upon significant change

29.5 Blameless Post-Mortems

    Blameless Culture
    ──────────────────

    OLD (blame culture):
    "Who deployed the bad code?"
    "Why didn't you test this?"
    → Engineers hide mistakes
    → Same mistakes repeated
    → Fear of being blamed prevents honesty

    BLAMELESS (systems thinking):
    "What conditions allowed this error to happen?"
    "Why did our system allow this deployment?"
    "Why didn't our tests catch this?"
    → Engineers share honestly
    → Root causes fixed
    → System becomes more resilient

    Key insight: Humans make mistakes. Good systems are designed
    to catch and recover from human mistakes.
    Focus on SYSTEM FAILURES, not HUMAN FAILURES.

Post-Mortem Template

# Post-Mortem: [Incident Title]

**Date**: 2024-01-15
**Duration**: 47 minutes (14:35 - 15:22 UTC)
**Severity**: SEV 2
**Incident Commander**: Alice
**Authors**: Alice, Bob, Carol

---

## Summary

Brief paragraph describing what happened, who was affected,
and what was done to resolve it.

The checkout service experienced a 47-minute partial outage
affecting 30% of users. Root cause was a misconfigured database
connection pool size deployed at 14:30 UTC that caused connection
exhaustion under normal load.

---

## Impact

| Metric | Value |
|--------|-------|
| Duration | 47 minutes |
| Users affected | ~15,000 (30% of concurrent users) |
| Transactions failed | ~4,500 |
| Revenue impact | ~$67,000 |
| Error budget consumed | 22 of 43 minutes (51% of monthly budget) |

---

## Timeline

| Time (UTC) | Event |
|-----------|-------|
| 14:30 | Deployment of v2.3.4 completed |
| 14:35 | First alert: checkout error rate > 5% |
| 14:38 | IC declared, @alice assigned |
| 14:45 | Status page updated |
| 14:50 | DB connection pool exhaustion identified in logs |
| 15:05 | Decision to roll back v2.3.4 |
| 15:18 | Rollback completed |
| 15:22 | Error rate back to normal, incident resolved |
| 15:25 | Status page updated: resolved |

---

## Root Cause Analysis

### Immediate Cause
DB connection pool max_connections was changed from 100 to 10
in the deployment config (typo: intended 100, wrote 10).

### Contributing Factors
1. **No validation**: Connection pool configuration not validated
   in deployment pipeline
2. **No canary**: Full deployment without gradual rollout
3. **No smoke test**: Post-deploy smoke test doesn't test DB connections
4. **Alert threshold**: DB connection alert threshold was too high (90%)
   — should have caught exhaustion earlier

### 5 Whys Analysis
1. Why did checkout fail? → DB connections exhausted
2. Why were connections exhausted? → Pool max_connections = 10 (too low)
3. Why was it set to 10? → Typo in deployment config change
4. Why wasn't the typo caught? → No validation for this config value
5. Why is there no validation? → Config validation isn't part of our deploy pipeline

**Root Cause**: Missing config validation in deployment pipeline allowed
an invalid connection pool size to reach production.

---

## What Went Well
- Alert fired within 5 minutes of impact starting
- IC was declared quickly and roles assigned
- Root cause identified within 15 minutes
- Rollback decision was made quickly without bureaucracy
- Status page was updated promptly

## What Went Poorly
- No canary deployment increased blast radius
- DB connection alert threshold was too high (missed early warning)
- Took 15 minutes to connect deployment to DB issue
- Rollback took 13 minutes (should be faster)

---

## Action Items

| Action | Owner | Due | Status |
|--------|-------|-----|--------|
| Add validation for DB pool size (1-500 range) | @carol | 2024-01-22 | ☐ |
| Implement canary deployments for checkout service | @platform | 2024-02-01 | ☐ |
| Lower DB connection alert threshold to 70% | @alice | 2024-01-17 | ☐ |
| Add post-deploy smoke test for DB connectivity | @bob | 2024-01-22 | ☐ |
| Reduce rollback time target to < 5 minutes | @platform | 2024-02-01 | ☐ |

---

*Post-mortem reviewed and approved by: CTO, Engineering Manager*

29.6 On-Call Runbooks

# Runbook: High DB Connection Count

## Alert
`alert: HighDBConnectionCount | severity: warning`

## Symptoms
- Checkout service returning 503 errors
- DB connection count > 80% of pool max

## Quick Triage (First 5 minutes)

1. Check current connection count:
   ```sql
   SELECT count(*) FROM pg_stat_activity WHERE datname = 'checkout';

Check what’s holding connections:

SELECT pid, now() - pg_stat_activity.query_start AS duration,
       query, state
FROM pg_stat_activity
WHERE state != 'idle' AND datname = 'checkout'
ORDER BY duration DESC LIMIT 20;

Check recent deployments:

kubectl rollout history deployment/checkout-service

Resolution Steps

If connections are idle (connection leak):

# Kill idle connections older than 10 minutes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND query_start < NOW() - INTERVAL '10 minutes'
  AND datname = 'checkout';

If pool is exhausted (under load):

# Emergency: increase pool size temporarily
kubectl set env deployment/checkout-service DB_POOL_MAX=200

# Verify new pods come up
kubectl rollout status deployment/checkout-service

If caused by a slow query:

-- Find and kill the slow query
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE query_start < NOW() - INTERVAL '5 minutes';

Escalation

If not resolved in 15 minutes: escalate to @dba-oncall
If data corruption suspected: escalate to SEV 1, page @cto

Prevention Notes

DB_POOL_MAX should be between 10 and 500
Each instance uses ~20 connections under normal load
5 instances × 20 = 100 connections minimum

---

## 29.7 Interview Questions

**Q1: What does "blameless post-mortem" mean and why is it important?**

> **Answer**: A blameless post-mortem focuses on identifying and fixing the system conditions that allowed an incident to occur, rather than identifying and punishing the individual who made a mistake. The key insight is that humans make errors, and good systems are designed to catch and tolerate human mistakes. If post-mortems are blame-oriented, engineers learn to hide mistakes, cover their tracks, and avoid taking risks — leading to a culture of fear and repeated incidents. Blameless post-mortems: (1) Encourage honest timelines (nobody hides their actions). (2) Surface systemic problems (missing tests, alerts, validation). (3) Lead to actual improvements. (4) Build psychological safety. Google's SRE Book describes this: "The goal of blameless post-mortems is not to avoid pointing out individual mistakes, but to identify the conditions that allowed mistakes to happen."

**Q2: What is MTTD, MTTA, and MTTR? Which is most important?**

> **Answer**: MTTD (Mean Time to Detect): How long from an incident starting until we become aware of it. MTTA (Mean Time to Acknowledge): How long from an alert firing until someone responds. MTTR (Mean Time to Resolve/Recover): How long from incident detection until service is restored. All three matter, but reducing MTTD and MTTA have the most leverage because they gate everything else. An incident you don't detect until users complain has already caused maximum damage. MTTD is reduced by better monitoring and alerting coverage. MTTA is reduced by on-call rotation, runbooks, and clear escalation paths. MTTR is reduced by runbooks, canary deployments (easy rollback), and practiced incident drills.

---

## 29.8 Summary

| Concept | Key Point |
|---------|-----------|
| Incident roles | IC (decide), Comms (communicate), SME (fix), Scribe (document) |
| Blameless PM | Focus on system failures, not human errors |
| 5 Whys | Root cause analysis technique |
| Action items | Post-mortem only valuable if actions are completed |
| Runbooks | Pre-written response guides for known alert conditions |
| Error budget | Incident cost measured in budget consumption |

---

*Next Chapter: [Chapter 30: Chaos Engineering](./30_chaos_engineering)*
---

## War Rooms

A **War Room** (physical or virtual) is a centralized communication channel established during a major incident. It ensures all relevant responders are in one place, reducing communication latency.

### War Room Best Practices
- **Dedicated Bridge**: Use a dedicated Zoom/Meet link or Slack channel (e.g., `#inc-20261027-db-down`).
- **Strict Roles**: The Incident Commander (IC) controls the room. Observers remain silent.
- **Single Source of Truth**: Pin a live document (Google Doc) containing the incident timeline, current status, and theories being tested.
- **Periodic Updates**: The IC or Communications Lead broadcasts status updates every 15-30 minutes to stakeholders outside the war room.

---

## Prerequisites

Chapter 41 (SRE Principles).

---

## Advantages & Disadvantages

**Advantages:** Brings order to chaos, strictly defined roles speed up resolution (MTTR).
**Disadvantages:** Requires rigorous training; blameless culture is hard to achieve in corporate environments.

---

## Common Mistakes

- Having no clear Incident Commander, leading to chaos and duplicated effort.
- Blaming individuals during the post-mortem, preventing honest analysis.
- Failing to communicate with stakeholders during the incident.

---

## Troubleshooting

| Symptom | Cause | Diagnosis | Fix |
|---------|-------|-----------|-----|
| Incident takes too long to resolve | Lack of runbooks or unclear roles | `Review MTTR metrics` | Create runbooks; train team on incident response roles (ICS) |
| Same incident happens twice | Action items from post-mortem were ignored | `Review past post-mortems` | Prioritize post-mortem action items in the engineering backlog |

---

## Hands-on Lab

**Objective**: Draft an incident timeline.
```bash
# 1. Extract logs for a specific time window to build a timeline
journalctl --since "2023-10-27 10:00:00" --until "2023-10-27 11:00:00" > incident_timeline.txt

Exercises

Draft a Blameless Post-Mortem for a recent (even minor) issue you faced, using the 5 Whys technique.
Define the roles for your ideal Incident Response team (Commander, Communicator, Operations).

Revision Notes

Incident Command System (ICS) provides a clear structure during chaos.
MTTR (Mean Time To Recovery) is a key metric for incident response.
Post-mortems must be blameless and focus on systemic improvements.
War Rooms (virtual or physical) are essential for centralized communication during major incidents.