SLI, SLO, SLA, RED, USE & Four Golden Signals

Chapter 30: SLI, SLO, SLA, RED, USE & Four Golden Signals

Learning Objectives

Define and measure SLIs for different service types
Set meaningful SLOs and derive error budgets
Apply the RED, USE, and Four Golden Signals methods
Design alerting strategies that don’t cause fatigue

What is this? (Beginner On-Ramp)

SLIs, SLOs, and SLAs are how engineering teams measure and promise reliability. An SLI is the actual measurement (e.g., 99% uptime). An SLO is the internal goal (aiming for 99.9%). An SLA is the legal contract promising money back if the goal isn’t met.

30.1 The Four Golden Signals (Google SRE)

    Four Golden Signals — Monitor These for Any Service
    ─────────────────────────────────────────────────────

    1. LATENCY: Time to serve a request
       • Track both successful and failed request latency
       • Failed requests often fail quickly → skew averages down
       • Measure: P50, P95, P99, P999
       • Never just average!

    2. TRAFFIC: Demand on the system
       • HTTP RPS, gRPC calls/sec, DB transactions/sec
       • Queue depth, active connections
       • Used for: capacity planning, anomaly detection

    3. ERRORS: Rate of failed requests
       • HTTP 5xx rate
       • Timeouts, application exceptions
       • Explicitly tracked + implicit (latency > threshold = error)

    4. SATURATION: How "full" is the service?
       • CPU %, memory %, disk %, queue depth
       • Latency often correlates with saturation
       • Predict problems before users notice

    These four signals cover 95% of what you need to monitor.
    If all four are green, your service is probably healthy.

30.2 RED Method (Services)

    RED Method: For REQUEST-DRIVEN SERVICES
    ────────────────────────────────────────

    R = Rate:      Requests per second
    E = Errors:    Error rate (failed requests / total)
    D = Duration:  Latency distribution (P99)

    PromQL for RED:

# Rate (requests/second)
sum(rate(http_requests_total[5m])) by (service)

# Errors (error rate %)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service) * 100

# Duration (P99 latency)
histogram_quantile(0.99,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

30.3 USE Method (Resources)

    USE Method: For INFRASTRUCTURE RESOURCES
    ─────────────────────────────────────────

    U = Utilization:  % time resource is busy
    S = Saturation:   Work queuing up (can't be served immediately)
    E = Errors:       Error events

    For every resource:
    CPU:
    U = 1 - idle% (from node_cpu_seconds_total)
    S = load_avg / nproc (scheduler run queue)
    E = hardware errors (MCE, etc.)

    Memory:
    U = (total - available) / total
    S = swap activity (si/so in vmstat)
    E = OOM killer events

    Disk:
    U = %util (iostat)
    S = await, aqu-sz (iostat)
    E = I/O errors (dmesg, SMART)

    Network:
    U = bytes / link_capacity
    S = dropped packets, send queue overflows
    E = interface errors

30.4 SLI/SLO/SLA (Full Reference)

    Availability SLI Examples by Service Type
    ───────────────────────────────────────────

    HTTP API:
    SLI = successful_requests / total_requests
    Good event: HTTP response in < 200ms with status 2xx or 3xx
    Bad event:  HTTP 5xx, timeout, or > 200ms

    Data Pipeline:
    SLI = records_processed_without_error / total_records
    Good event: Record processed within 5 minutes of arrival
    Bad event:  Processing error OR freshness > 5 minutes

    Database:
    SLI = successful_queries / total_queries
    Good event: Query returns result in < 50ms
    Bad event:  Query error or latency > 50ms

    Object Storage:
    SLI = objects_durably_stored / objects_written
    Good event: Object stored and readable after write
    Bad event:  Object lost, corrupted, or unreadable

    Typical SLO Targets:
    ──────────────────────
    Internal API:         99.5%  (3.6 hours/month downtime)
    External API (basic): 99.9%  (43 minutes/month)
    External API (SLA):   99.95% (22 minutes/month)
    Critical payment:     99.99% (4 minutes/month)
    Five nines (rare!):   99.999% (26 seconds/month — very expensive)

30.5 Alert Design Best Practices

    Good vs Bad Alerts
    ───────────────────

    BAD Alert: CPU > 90%
    → Fires constantly during normal workloads
    → No context on whether users are affected
    → Alert fatigue: team starts ignoring alerts

    GOOD Alert: Error budget burn rate > 14x
    → Directly tied to user impact (SLO)
    → Fires only when something significant is happening
    → Actionable: runbook says what to check

    Alert Design Principles:
    ─────────────────────────
    1. Every alert should be actionable
       If you can't do anything about it → not an alert, use a dashboard

    2. Alert on symptoms, not causes
       Alert: "High P99 latency" (symptom)
       Not: "CPU > 80%" (cause — might not affect users)

    3. Use "for" clause to avoid spurious alerts
       for: 5m → must be continuously true for 5 minutes

    4. Have runbooks for every alert
       Alert fires → engineer knows exactly what to do
       No runbook → unclear alert, don't page someone

    5. Alert on burn rate, not instantaneous values
       Error rate briefly spikes to 5% → might not matter
       Error rate sustained at 1% for 6 hours → eating the budget

30.6 Interview Questions

Q1: What are the Four Golden Signals and why are they sufficient for most services?

Answer: The Four Golden Signals (from Google SRE Book) are: (1) Latency — time to serve a request (track P99, not just average; track errors separately since failed requests are often fast). (2) Traffic — demand on the system (RPS, connections, queue depth). (3) Errors — rate of failed requests (5xx, timeouts, explicit exceptions). (4) Saturation — how “full” is the system (CPU %, memory %, queue depth, connection pool usage). They’re sufficient because: Latency tells you if users are experiencing slowness. Errors tell you if requests are failing. Traffic tells you demand and anomalies. Saturation predicts future problems before they impact users. Together they give a complete picture of service health from both user experience and infrastructure perspectives.

Q2: What is the RED method and how does it differ from USE?

Answer: RED (Rate, Errors, Duration) is for request-driven services — APIs, microservices, databases. USE (Utilization, Saturation, Errors) is for infrastructure resources — CPUs, memory, disks, network. RED answers “is this service serving users well?” USE answers “is this resource healthy?” Use RED to detect user-facing problems, then use USE to find the infrastructure root cause. Example: RED shows high P99 latency on the checkout API. USE shows disk saturation on the database node. Drill into USE metrics to confirm the database disk is the root cause.

30.7 Summary

Method	Applies To	Metrics
4 Golden Signals	Any service	Latency, Traffic, Errors, Saturation
RED	Request services	Rate, Errors, Duration
USE	Infrastructure	Utilization, Saturation, Errors
SLI	Service health	Quantitative metric
SLO	Internal target	SLI threshold
SLA	External contract	Legal commitment

Next Chapter: Chapter 31: Alert Design & Dashboard Design

Prerequisites

Chapter 28 (Observability).

Advantages & Disadvantages

Advantages: Aligns engineering with business goals, Error Budgets depersonalize release decisions. Disadvantages: Hard to define accurate SLIs that truly reflect user experience.

Common Mistakes

Setting SLOs to 100% — mathematically impossible and discourages innovation.
Measuring SLIs from the server side instead of the client side (e.g., measuring at the DB instead of the load balancer).

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Error budget consumed rapidly	Sustained minor outage or unrealistic SLO	`Review incident timeline and SLO definition`	Halt feature releases; focus on reliability; or renegotiate the SLO if it’s too strict

Hands-on Lab

Objective: Calculate an SLI from Prometheus metrics.

# Example PromQL for an Availability SLI (Good events / Total events)
# sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Exercises

Define an SLI, SLO, and SLA for a hypothetical e-commerce checkout service. What are the consequences of breaching the SLA?
Calculate the error budget in minutes for a 99.9% availability SLO over a 30-day window.

Revision Notes

SLI = Measurement (e.g., 99.5% successful requests).
SLO = Target (e.g., We aim for 99.9% success).
SLA = Contract (e.g., We pay you if we drop below 99.5%).
Error Budgets = 100% - SLO. Used to balance feature velocity with reliability.