Metrics, Logs & Traces: The Three Pillars

Chapter 28: Metrics, Logs & Traces: The Three Pillars of Observability

Learning Objectives

• Understand the three pillars and when to use each
• Configure Prometheus with PromQL for metrics
• Build a centralized logging pipeline with ELK/EFK
• Set up distributed tracing with OpenTelemetry

---

What is this? (Beginner On-Ramp)

Observability is how we understand what’s happening inside complex software. We use three tools: Metrics (numbers and charts showing health), Logs (text records of events), and Traces (maps showing the exact path a request took through the system).

28.1 The Three Pillars

    Observability = Metrics + Logs + Traces
    ─────────────────────────────────────────

    METRICS: Numerical measurements over time
    "What is happening?" (aggregated, efficient)
    Examples: CPU %, RPS, P99 latency, error rate
    Tools: Prometheus, InfluxDB, Datadog
    Best for: Alerting, dashboards, capacity planning

    LOGS: Timestamped records of events
    "What happened?" (detailed, verbose)
    Examples: "User alice logged in from 1.2.3.4 at 10:23:45"
    Tools: ELK, Loki, Splunk
    Best for: Debugging specific incidents, audit trails

    TRACES: Request journeys across services
    "How did it happen?" (per-request, correlated)
    Examples: checkout → payment → stripe API (2.1s total)
    Tools: Jaeger, Tempo, Zipkin, OpenTelemetry
    Best for: Latency root cause, service dependency mapping

    Use Together:
    ─────────────
    Alert fires (metrics) → Look at logs (what happened?)
    → Jump to trace (which service/call was slow?)
    → Metrics confirm fix (is P99 back to normal?)

---

28.2 Pillar 1: Metrics — Prometheus

Architecture

    ┌──────────────────────────────────────────────────────────┐
    │  TARGETS (Exporters)                                      │
    │  node_exporter :9100  nginx_exporter :9113  app :8080    │
    └───────────────────────────────┬──────────────────────────┘
                                    │ HTTP GET /metrics (pull)
    ┌───────────────────────────────▼──────────────────────────┐
    │  PROMETHEUS SERVER                                        │
    │  Scrape Engine → TSDB (15d retention) → Alert Eval       │
    │  HTTP API /api/v1/query → PromQL                         │
    └──────────────────────┬───────────────────────────────────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
         Grafana                  Alertmanager
         (Dashboards)             (PagerDuty/Slack)

Key PromQL Queries

# RED Method
sum(rate(http_requests_total[5m])) by (service)            # Rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))  # Errors
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # Duration

# USE Method
(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100    # CPU utilization
(1 - node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes) * 100  # Memory utilization
(1 - node_filesystem_avail_bytes/node_filesystem_size_bytes) * 100    # Disk utilization

# Predict disk full
predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 4*3600) < 0

Alertmanager Config (Production)

route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  repeat_interval: 4h
  receiver: 'slack-warnings'
  routes:
    - match: { severity: critical }
      receiver: 'pagerduty'
    - match: { severity: warning }
      receiver: 'slack-warnings'

receivers:
  - name: pagerduty
    pagerduty_configs:
      - routing_key: 'YOUR_PD_KEY'
  - name: slack-warnings
    slack_configs:
      - channel: '#alerts'
        send_resolved: true

---

28.3 Pillar 2: Logs — ELK/EFK Stack

The Logging Scale Problem

    100 servers × 10 services × 1000 logs/sec = 1M logs/sec
    = 86 billion logs/day

    Problems with per-server logs:
    ✗ Can't search across all servers
    ✗ Logs lost when server decommissioned
    ✗ No cross-service correlation
    ✗ SSH to each server for debugging

    Solution: Centralized pipeline
    Applications → Shipper → Buffer → Parser → Storage → UI
    (stdout)     (Filebeat)  (Kafka) (Logstash) (ES)   (Kibana)

EFK Stack Architecture

    ┌─────────────────────────────────────────────────────────┐
    │  Applications / Servers (nginx, app, postgres, system)  │
    └───────────────────────────────┬─────────────────────────┘
                                    │ DaemonSet / per-host
                                    ▼
                              ┌──────────────┐
                              │  Fluent Bit  │  ~5MB RAM
                              │  (per node)  │  Parse + tag
                              └──────┬───────┘
                                     │
                                     ▼
                              ┌──────────────┐
                              │    Kafka     │  Buffer for spikes
                              └──────┬───────┘
                                     │
                                     ▼
                              ┌──────────────┐
                              │  Logstash /  │  Enrich, GeoIP
                              │   Fluentd    │  Parse access logs
                              └──────┬───────┘
                                     │
                                     ▼
                              ┌──────────────┐
                              │Elasticsearch │  Hot→Warm→Cold ILM
                              └──────┬───────┘
                                     │
                                     ▼
                              ┌──────────────┐
                              │    Kibana    │  Search, alert
                              └──────────────┘

Structured Logging (Always Use This)

    Unstructured (BAD):
    2024-01-15 10:23:45 [ERROR] User alice failed to login from 192.168.1.5
    → Requires fragile regex to extract fields

    Structured JSON (GOOD):
    {"timestamp":"2024-01-15T10:23:45Z","level":"error","user":"alice",
     "source_ip":"192.168.1.5","error_code":"AUTH_INVALID_PASSWORD","trace_id":"abc123"}
    → All fields directly indexable and searchable

# Python structured logging
import structlog
logger = structlog.get_logger()
logger.error("login_failed", user="alice", source_ip="1.2.3.4", attempts=3)

Fluent Bit Config (Kubernetes)

[INPUT]
    Name          tail
    Tag           kube.*
    Path          /var/log/containers/*.log
    Parser        docker
    Mem_Buf_Limit 50MB

[FILTER]
    Name          kubernetes
    Match         kube.*
    Merge_Log     On

[OUTPUT]
    Name          es
    Match         *
    Host          elasticsearch.logging.svc.cluster.local
    Port          9200
    Logstash_Format On
    tls           On

Elasticsearch ILM (Hot → Warm → Cold → Delete)

# Create lifecycle policy
curl -X PUT "http://es:9200/_ilm/policy/logs-policy" -H 'Content-Type: application/json' -d'{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": {"max_size":"50GB","max_age":"1d"} } },
      "warm":   { "min_age":"7d",  "actions": { "forcemerge":{"max_num_segments":1} } },
      "cold":   { "min_age":"30d", "actions": {} },
      "delete": { "min_age":"90d", "actions": { "delete": {} } }
    }
  }
}'

KQL Queries (Kibana)

service: nginx AND level: error
http.response.status_code >= 500
(level: error OR level: critical) AND NOT service: celery-worker
event.action: "user_login" AND event.outcome: "failure"

---

28.4 Pillar 3: Traces — OpenTelemetry

Context Propagation

    Distributed Trace
    ──────────────────
    TraceID: abc123 (same across all services)

    api-gateway  [████████████████████] 200ms
    └─ checkout     [████] 80ms
    └─ payment         [██████████] 100ms  ← slow!
       └─ stripe-api      [████████] 90ms  ← root cause

OTel Instrumentation (Python)

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="http://otel-collector:4317"
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        # ... business logic ...

---

28.5 Correlating All Three Pillars

    Workflow: Alert → Logs → Trace
    ────────────────────────────────

    1. Prometheus alert: P99 latency > 1s for checkout-service
    2. Grafana: drill into time range, see spike at 14:35
    3. Kibana: filter logs by service=checkout AND time=14:35
       → Log shows: "Payment timeout after 800ms, trace_id=abc123"
    4. Jaeger: open trace abc123
       → payment-service→stripe-api span took 750ms
    5. Root cause: Stripe API degradation

    Key: Always inject trace_id into every log line!

---

28.6 Interview Questions

Q1: What is the difference between metrics, logs, and traces?

Answer: Metrics are numerical measurements aggregated over time (CPU %, RPS, error rate) — efficient, always on, great for alerting and dashboards. They answer “what is happening?” Logs are timestamped textual records of discrete events — verbose, detailed, great for debugging “what happened?” Traces track individual requests end-to-end across multiple services — great for “how did this specific request behave and where was it slow?” The three pillars work together: an alert (metrics) triggers investigation → logs reveal details → traces show the request path that caused the issue.

Q2: What is structured logging and why is it critical at scale?

Answer: Structured logging outputs logs as machine-parseable JSON instead of free-form text. Each field (timestamp, level, service, user, trace_id, error_code) is a discrete key-value pair. At scale this is critical because: (1) Search: user: alice AND level: error instead of grep. (2) Aggregation: Count errors by service without regex. (3) Correlation: trace_id field links logs to traces. (4) Alerting: Alert on error_code: PAYMENT_FAILED > 10/min. Without structure, you’d need fragile regex to extract any value from logs.

---

28.7 Summary

Pillar	Tool	Use Case
Metrics	Prometheus + Grafana	Alerting, dashboards
Logs	Fluent Bit + Elasticsearch + Kibana	Debugging, audit
Traces	OpenTelemetry + Jaeger/Tempo	Latency root cause

---

Next Chapter: Chapter 29: OpenTelemetry & Distributed Tracing

Prerequisites

Basic understanding of distributed systems.

---

Advantages & Disadvantages

Advantages: Provides complete visibility into system health, crucial for microservices. Disadvantages: Massive storage costs for logs/traces, “alert fatigue” if poorly configured.

---

Common Mistakes

• Logging everything at DEBUG level in production — fills disks and creates impossible-to-search noise.
• Using metrics for high-cardinality data (like User IDs) — this causes TSDB (Prometheus) memory exhaustion.
• Treating traces as logs — traces show the path and timing of a request, logs show discrete events.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Prometheus running out of memory (OOM)	High cardinality metrics (too many labels)	promql: topk(10, count by (__name__) ({__name__=~'.+'}))	Drop high cardinality labels via metric_relabel_configs
Logs taking up too much disk space	Log rotation not configured or DEBUG level enabled	du -sh /var/log/*	Configure logrotate and set app log level to WARN/ERROR

---

Hands-on Lab

Objective: Query the three pillars.

# 1. Query metrics (PromQL)
# curl -g 'http://localhost:9090/api/v1/query?query=up'

# 2. Search logs (systemd journal)
journalctl -u nginx --since "1 hour ago" -p err

# 3. View traces (Jaeger/Tempo - assumed UI)

---

Exercises

1. Write a PromQL query that calculates the 99th percentile response time for a web service over the last 5 minutes.
2. Configure logrotate to rotate a custom application’s log file daily, keep 7 days of logs, and compress old logs.

---

Revision Notes

• Metrics (Numeric, aggregatable), Logs (Discrete events, detailed), Traces (Request path across services).
• High cardinality is the enemy of metrics systems.
• Structured logging (JSON) is required for modern centralized logging systems.

---

Further Reading

• Observability Engineering by Charity Majors
• Prometheus Documentation

---

Related Chapters

• Chapter 29 — OpenTelemetry for Tracing

---

Last Updated: July 2026