Skip to content
Flowchart Learning

Alert Design & Dashboard Design

Chapter 31: Alert Design & Dashboard Design

Section titled “Chapter 31: Alert Design & Dashboard Design”

Learning Objectives

Section titled “Learning Objectives”
  • Design alerts that are actionable and avoid alert fatigue
  • Build dashboards following proven patterns
  • Implement multi-window burn rate alerting
  • Structure dashboard hierarchy for different audiences

What is this? (Beginner On-Ramp)

Section titled “What is this? (Beginner On-Ramp)”

Alert and dashboard design is about creating effective warning systems. Good design ensures that engineers only get paged when something is actually broken and impacting users, and that dashboards clearly show exactly what is wrong without overwhelming them with data.

31.1 Alert Fatigue: The Production Problem

Section titled “31.1 Alert Fatigue: The Production Problem”
    Alert Fatigue Cycle
    ─────────────────────


    Too many noisy alerts
           │
           ▼
    On-call engineers start ignoring alerts
           │
           ▼
    Real incident fires → alert ignored
           │
           ▼
    Outage discovered by customers
           │
           ▼
    "Why didn't we catch this?" → Add more alerts!
           │
           └─── CYCLE REPEATS ──────────────────┐
                                                 │
    Break the cycle:                             │
    • Remove alerts that don't need action  ◄────┘
    • Require runbook for every alert
    • Use burn rate alerts (symptom-based)
    • Weekly alert review: remove spurious

31.2 Multi-Window Burn Rate Alerting

Section titled “31.2 Multi-Window Burn Rate Alerting”

This is the gold standard for production SLO alerting (from Google SRE Workbook):

/etc/prometheus/rules/slo_burn_rate.yml
groups:
  - name: slo-burn-rates
    rules:


      # ── Page: Fast burn (budget exhausted in < 2 days) ──
      - alert: SLOBurnRateFast
        expr: |
          (
            job:slo_error_ratio:rate1h > (14.4 * 0.001)
            AND
            job:slo_error_ratio:rate5m > (14.4 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          window: short
        annotations:
          summary: "Fast burn: {{ $labels.job }} will exhaust monthly error budget in ~2 days"


      # ── Ticket: Slow burn (budget exhausted in < 10 days) ─
      - alert: SLOBurnRateSlow
        expr: |
          (
            job:slo_error_ratio:rate6h > (6 * 0.001)
            AND
            job:slo_error_ratio:rate30m > (6 * 0.001)
          )
        for: 15m
        labels:
          severity: ticket
          window: long
        annotations:
          summary: "Slow burn: {{ $labels.job }} will exhaust monthly error budget in ~10 days"


  - name: slo-recording-rules
    rules:
      # Pre-compute error ratios for efficiency
      - record: job:slo_error_ratio:rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))


      - record: job:slo_error_ratio:rate30m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total[30m]))


      - record: job:slo_error_ratio:rate1h
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total[1h]))


      - record: job:slo_error_ratio:rate6h
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total[6h]))

31.3 Alert Runbook Template

Section titled “31.3 Alert Runbook Template”

Every alert must have a runbook. Here’s the template:

# Runbook: SLOBurnRateFast


## What This Alert Means
The service {{ $labels.job }} is consuming its monthly SLO error budget
at > 14x the normal rate. At this rate, the budget will be exhausted
in approximately 2 days.


SLO: 99.9% success rate
Error Budget: 43.2 minutes/month


## Impact
Users are experiencing errors or high latency above the SLO threshold.


## Immediate Steps (First 5 Minutes)
1. Check which specific errors are occurring:

sum by (status) (rate(http_requests_total[5m]))

2. Check if a recent deployment is correlated:
```bash
kubectl rollout history deployment/{{ $labels.job }}
  1. Check error logs:
    Terminal window
    kubectl logs -l app={{ $labels.job }} --since=10m | grep ERROR

Triage Decision Tree

Section titled “Triage Decision Tree”
  • If error rate > 50%: Declare SEV 1, page IC
  • If error rate 5-50%: Declare SEV 2, investigate
  • If correlated with deploy: Rollback immediately
  • If not correlated: Check dependencies (DB, cache, upstream)

Resolution Paths

Section titled “Resolution Paths”
  • Deploy rollback: kubectl rollout undo deployment/{{ $labels.job }}
  • Scale up: kubectl scale deployment/{{ $labels.job }} --replicas=10
  • Circuit break: Disable the feature flag for the failing feature

Escalation

Section titled “Escalation”
  • Not resolved in 30 min: Escalate to service team lead
  • If DB involved: Page DBA on-call
---


## 31.4 Dashboard Hierarchy
Three-Tier Dashboard Design
────────────────────────────


Tier 1: Executive/Business Dashboard
────────────────────────────────────
• Audience: Leadership, product, business
• Refresh: 5 minutes
• Content: Overall health, SLO compliance, revenue impact
• Time range: 30 days


[99.97% Availability] [12ms P99] [0.02% Error Rate] [SLO: ✅]
[Orders/hour trend] [Revenue trend]


Tier 2: Service Dashboard
──────────────────────────
• Audience: On-call engineers, service owners
• Refresh: 30 seconds
• Content: RED metrics, SLO burn rate, recent deployments
• Time range: 24 hours


[Request Rate] [Error Rate] [P99 Latency] [Error Budget Remaining]
[Per-endpoint breakdown] [Dependency health]


Tier 3: Debug/Drill-Down Dashboard
────────────────────────────────────
• Audience: Engineers investigating a specific incident
• Refresh: 10 seconds
• Content: All metrics, logs, traces links
• Time range: Last 1-2 hours


[Per-pod CPU/Memory] [GC pressure] [DB connection pool]
[Query latency histograms] [Log error count by type]
---


## 31.5 Grafana Dashboard Best Practices


```json
// Grafana dashboard tips (JSON config fragments)


// 1. Always use variables for environment/cluster
{
  "templating": {
    "list": [
      {
        "name": "environment",
        "type": "custom",
        "options": [
          {"text": "production", "value": "production"},
          {"text": "staging", "value": "staging"}
        ]
      },
      {
        "name": "service",
        "type": "query",
        "query": "label_values(http_requests_total, job)"
      }
    ]
  }
}


// 2. Use stat panels with thresholds for SLO status
{
  "type": "stat",
  "title": "Availability SLO",
  "targets": [{
    "expr": "(1 - sum(rate(http_requests_total{status=~'5..', job='$service'}[30d])) / sum(rate(http_requests_total{job='$service'}[30d]))) * 100"
  }],
  "fieldConfig": {
    "defaults": {
      "unit": "percent",
      "thresholds": {
        "steps": [
          {"value": 0, "color": "red"},
          {"value": 99.5, "color": "yellow"},
          {"value": 99.9, "color": "green"}
        ]
      }
    }
  }
}
    Dashboard Layout Pattern (Service Dashboard)
    ──────────────────────────────────────────────


    Row 1: SLO Status (stat panels)
    ┌─────────────────┬──────────────┬──────────────┬───────────────┐
    │  Availability   │  P99 Latency │  Error Rate  │ Budget Left   │
    │  99.97% ✅      │  45ms ✅     │  0.03% ✅    │  87% (37min)  │
    └─────────────────┴──────────────┴──────────────┴───────────────┘


    Row 2: Traffic (time series, 24h)
    ┌─────────────────────────────────────────────────────────────────┐
    │  Request Rate (RPS) — by status code                           │
    │  [████ 2xx ████████████████████████████████████████████████]   │
    │  [                                                    5xx ██]  │
    └─────────────────────────────────────────────────────────────────┘


    Row 3: Latency (heatmap or time series)
    ┌───────────────────────────────────┬────────────────────────────┐
    │  P50 / P95 / P99 Latency          │  Latency Heatmap           │
    └───────────────────────────────────┴────────────────────────────┘


    Row 4: Resources
    ┌───────────────┬──────────────────┬─────────────────────────────┐
    │  CPU Usage    │  Memory Usage    │  DB Connection Pool         │
    └───────────────┴──────────────────┴─────────────────────────────┘

31.6 Interview Questions

Section titled “31.6 Interview Questions”

Q1: What is alert fatigue and how do you combat it?

Answer: Alert fatigue occurs when engineers receive so many alerts (many false positives or non-actionable) that they start ignoring them — including real incidents. Combat it by: (1) Every alert must be actionable — if there’s no action to take, it’s not an alert, it’s a dashboard. (2) Alert on symptoms, not causes — “error rate high” (symptom) not “CPU high” (cause that may not affect users). (3) Burn rate alerting instead of instantaneous thresholds — catches sustained issues not brief spikes. (4) “for” clause in Prometheus rules — require the condition to persist for 5+ minutes. (5) Weekly alert review — look at which alerts fired last week, remove noisy ones. (6) On-call metrics — track MTTA and alert volume; if >5 alerts/shift → too many.

Q2: Describe the structure of a good production dashboard.

Answer: A good production service dashboard has: (1) Top row: SLO status — single-stat panels showing current availability, P99 latency, error rate, and error budget remaining. Color-coded green/yellow/red. At a glance, on-call knows if anything is wrong. (2) Middle rows: RED metrics as time-series graphs — request rate (by status code), error rate over time, latency P50/P95/P99. Time range: last 24 hours with zoom to last 1 hour. (3) Bottom rows: Resource saturation — CPU, memory, DB connections, queue depth. (4) Variables: Drop-downs for environment (prod/staging) and service. (5) Links: Each panel links to Loki logs or Jaeger traces filtered to the same time range.

31.7 Summary

Section titled “31.7 Summary”
ConceptKey Point
Alert fatigueToo many alerts → engineers ignore them
Symptom alertingAlert on user impact, not resource usage
Burn rate alertsMulti-window (1h+5m or 6h+30m)
RunbooksEvery alert must have documented response
Dashboard tiersExecutive → Service → Debug
VariablesEnvironment, service, time range selectors

Next Chapter: Chapter 32: Zero Trust Architecture

Section titled “Next Chapter: Chapter 32: Zero Trust Architecture”

Prerequisites

Section titled “Prerequisites”

Chapter 28 (Observability), Chapter 30 (SLIs/SLOs).

Advantages & Disadvantages

Section titled “Advantages & Disadvantages”

Advantages: Symptom-based alerts reduce noise, actionable dashboards speed up MTTR. Disadvantages: Dashboards require constant maintenance as architecture changes.

Common Mistakes

Section titled “Common Mistakes”
  • Alerting on causes (e.g., CPU > 80%) instead of symptoms (e.g., API latency > 2s).
  • Sending alerts to email (ignored) instead of a paging system (PagerDuty).
  • Creating dashboards with 100+ charts that no one looks at until an incident.

Troubleshooting

Section titled “Troubleshooting”
SymptomCauseDiagnosisFix
Alert fatigue (too many false positives)Thresholds too tight or alerting on non-actionable metricsReview alert historyTune thresholds, require duration (for: 5m), or route to ticketing instead of paging

Hands-on Lab

Section titled “Hands-on Lab”

Objective: Write a symptom-based alert in Prometheus.

# Example Prometheus Alert Rule
groups:
- name: API_Alerts
  rules:
  - alert: HighErrorRate
    expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "High 5xx error rate on {{ $labels.job }}"

Exercises

Section titled “Exercises”
  1. Redesign an alert that triggers when ‘Disk Space > 90%’ to instead trigger based on ‘Time until Disk Full < 4 hours’ using PromQL predict_linear.
  2. Design a ‘Golden Signals’ dashboard layout (Latency, Traffic, Errors, Saturation) for a web service.

Revision Notes

Section titled “Revision Notes”
  • Alert on Symptoms (user impact), not Causes (internal state).
  • Every pageable alert must be actionable.
  • Dashboards should be structured hierarchically: High-level (SLOs) -> Service level (Golden Signals) -> Infrastructure level.

Further Reading

Section titled “Further Reading”
  • Site Reliability Engineering - Chapter on Monitoring Distributed Systems
Section titled “Related Chapters”

Last Updated: July 2026