Distributed Tracing: OpenTelemetry & Jaeger

Chapter 29: Distributed Tracing: OpenTelemetry & Jaeger

Learning Objectives

• Understand distributed tracing concepts (traces, spans, context propagation)
• Know OpenTelemetry (OTel) architecture and instrumentation
• Configure Jaeger and Tempo as trace backends
• Correlate traces with logs and metrics

---

What is this? (Beginner On-Ramp)

OpenTelemetry is the modern, universal standard for collecting observability data. Instead of using a dozen different tools to gather metrics and logs, OpenTelemetry provides a single, unified way for applications to report on their own health and performance.

27.1 The Distributed Tracing Problem

    Why Distributed Tracing is Needed
    ────────────────────────────────────

    User request: POST /checkout

    Without Tracing (where is the 2-second latency?):
    ───────────────────────────────────────────────────
    [api-gateway]  received request
    [order-service] created order
    [payment-service] charged card
    [inventory-service] reserved items
    [notification-service] sent email
    [api-gateway] returned 200 OK  ← took 2 seconds total!

    With Distributed Tracing:
    ──────────────────────────
    Trace ID: abc123
    Total: 2000ms

    api-gateway        [████████████████████████████████] 2000ms
    ├─ order-service      [████] 400ms
    ├─ payment-service       [████████████████] 1400ms  ← HERE!
    │  ├─ fraud-check              [████] 400ms
    │  └─ stripe-api                   [████████] 900ms  ← External!
    └─ notification-service [██] 200ms

    Finding: payment-service → stripe-api is 900ms
    Action: Stripe API timeout, add circuit breaker

---

27.2 Tracing Concepts

    Core Concepts
    ──────────────

    Trace:   A complete end-to-end request across all services
             Identified by: TraceID (globally unique)

    Span:    A single unit of work within a trace
             Has: SpanID, ParentSpanID, name, start/end time, tags, logs

    Context: Metadata propagated between services in HTTP headers
             W3C Trace Context: traceparent: 00-traceid-spanid-flags

    Baggage: Key-value pairs propagated along with trace context
             Example: user.id, tenant.id, feature.flag

    Span Relationships:
    ─────────────────────
    Parent → Child: Sequential
    ChildOf:        Child depends on parent result
    FollowsFrom:    Child doesn't need parent result (async)

    Span Tags (important ones):
    ─────────────────────────────
    http.method, http.url, http.status_code
    db.system, db.statement
    error (boolean), error.message
    net.peer.name, net.peer.port

---

27.3 OpenTelemetry Architecture

    OpenTelemetry (OTel)
    ─────────────────────

    OTel is the vendor-neutral, CNCF standard for observability.
    Provides: SDK + APIs + Collector (for all 3 pillars: Metrics, Logs, Traces)

    ┌─────────────────────────────────────────────────────────┐
    │  Applications                                            │
    │  ┌──────────────────────────────────────────────────┐  │
    │  │  OTel SDK (auto-instrumentation or manual)        │  │
    │  │  Languages: Go, Python, Java, JS, .NET, Rust...  │  │
    │  └───────────────────────┬──────────────────────────┘  │
    └──────────────────────────│─────────────────────────────┘
                               │ OTLP (OpenTelemetry Protocol)
                               │ gRPC or HTTP
                               ▼
    ┌─────────────────────────────────────────────────────────┐
    │  OTel Collector (optional but recommended)               │
    │  ┌────────────┐ ┌────────────┐ ┌────────────────────┐  │
    │  │ Receivers  │ │ Processors │ │ Exporters          │  │
    │  │ OTLP gRPC  │ │ Batch      │ │ Jaeger             │  │
    │  │ OTLP HTTP  │ │ Sampling   │ │ Tempo              │  │
    │  │ Zipkin     │ │ Transform  │ │ Elasticsearch      │  │
    │  │ Prometheus │ │ Filter     │ │ Prometheus         │  │
    │  └────────────┘ └────────────┘ └────────────────────┘  │
    └─────────────────────────────────────────────────────────┘
                               │
            ┌──────────────────┼──────────────────┐
            ▼                  ▼                  ▼
        Jaeger/Tempo       Prometheus          Elasticsearch
        (Traces)           (Metrics)           (Logs)

---

27.4 OpenTelemetry Instrumentation

Auto-Instrumentation (Zero Code Change)

# Python auto-instrumentation
# pip install opentelemetry-distro opentelemetry-instrumentation-flask
# opentelemetry-bootstrap -a install

# Run with auto-instrumentation (no code changes!):
opentelemetry-instrument \
    --exporter_otlp_endpoint=http://otel-collector:4317 \
    --service_name=order-service \
    python app.py

# Auto-instruments: Flask/Django, SQLAlchemy, Redis, requests, etc.

Manual Instrumentation

# Manual OpenTelemetry instrumentation in Python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Setup
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Create spans
def process_order(order_id: str, user_id: str):
    with tracer.start_as_current_span("process_order") as span:
        # Add attributes to the span
        span.set_attribute("order.id", order_id)
        span.set_attribute("user.id", user_id)

        try:
            # Child span
            with tracer.start_as_current_span("validate_inventory") as child:
                child.set_attribute("db.system", "postgresql")
                result = db.query("SELECT * FROM inventory WHERE order_id = %s", order_id)

            with tracer.start_as_current_span("charge_payment") as payment_span:
                payment_span.set_attribute("payment.amount", order.total)
                response = payment_client.charge(order.total)
                payment_span.set_attribute("payment.status", response.status)

            span.set_attribute("order.status", "completed")
            return {"status": "ok"}

        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise

# HTTP context propagation (inject/extract headers)
# In outbound HTTP request:
from opentelemetry.propagators.b3 import B3MultiFormat
import requests

propagator = TraceContextTextMapPropagator()
headers = {}
propagator.inject(headers)  # Adds traceparent header

requests.post("http://payment-service/charge", json=data, headers=headers)

# In receiving service:
context = propagator.extract(request.headers)
with tracer.start_as_current_span("handle_charge", context=context) as span:
    pass

---

27.5 OTel Collector Configuration

/etc/otelcol/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Also receive Prometheus metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          static_configs:
            - targets: ['localhost:8888']

processors:
  # Batch spans before sending (efficiency)
  batch:
    timeout: 1s
    send_batch_size: 1024

  # Memory limiter (prevent OOM)
  memory_limiter:
    check_interval: 5s
    limit_mib: 512
    spike_limit_mib: 128

  # Add resource attributes
  resource:
    attributes:
      - key: environment
        value: production
        action: upsert

  # Tail-based sampling (sample only if there's an error or slow request)
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces-policy
        type: latency
        latency:
          threshold_ms: 1000
      - name: random-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 1  # 1% of everything else

exporters:
  # Send traces to Jaeger
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  # Send traces to Tempo (Grafana)
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Send metrics to Prometheus
  prometheus:
    endpoint: 0.0.0.0:8889

  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [jaeger, otlp/tempo]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

---

27.6 Trace Sampling Strategy

    Sampling Strategies
    ────────────────────

    Head-based Sampling (decide at start):
    ✓ Simple to implement
    ✗ May miss important traces (errors, slow requests)

    Example: Sample 1% of all traces
    → 99% of data is lost, including most errors!

    Tail-based Sampling (decide after completion):
    ✓ Can keep all errors, all slow requests
    ✓ Can keep 1% of normal requests
    ✗ Must buffer traces until decision is made (memory)
    ✗ More complex (need OTel Collector with state)

    Production Recommended Strategy:
    ──────────────────────────────────
    Always keep (100%): errors, latency > 1s, specific endpoints
    Sample (1-5%):      successful fast requests

    This ensures:
    • All problems are captured
    • Storage is manageable
    • Normal traffic is represented statistically

---

27.7 Correlating Traces, Metrics, Logs

    The Three Pillars of Observability (Connected)
    ────────────────────────────────────────────────

    Alert fires: High P99 latency on /checkout

    Step 1: Grafana metrics dashboard shows spike at 14:35

    Step 2: Click on trace ID in Grafana → Jaeger opens that specific trace

    Step 3: In Jaeger, see that payment-service span took 2000ms

    Step 4: Click "Logs" button → Kibana opens logs filtered by
            trace_id=abc123 AND service=payment-service

    Step 5: Logs show: "Stripe API timeout after 1500ms, retrying..."

    Step 6: Root cause found: Stripe API degradation

    Key: Inject trace_id into every log line

    Implementation:
    ────────────────
    # Add trace context to log (Python)
    import logging
    from opentelemetry import trace

    class TraceContextFilter(logging.Filter):
        def filter(self, record):
            span = trace.get_current_span()
            if span.is_recording():
                ctx = span.get_span_context()
                record.trace_id = format(ctx.trace_id, '032x')
                record.span_id = format(ctx.span_id, '016x')
            return True

---

27.8 Interview Questions

Q1: What is distributed tracing and what problem does it solve?

Answer: Distributed tracing tracks a single request as it flows through multiple microservices. Each service adds a “span” (recording what it did and how long it took), and all spans for one request are linked by a shared TraceID. This solves the “needle in a haystack” problem: when a request takes 3 seconds across 8 services, you can instantly see which service and which operation within it caused the latency. Without tracing, you’d grep logs across all 8 services, correlate timestamps manually, and still might not find the root cause if the slow operation isn’t logged.

Q2: What is OpenTelemetry and why has it become the standard?

Answer: OpenTelemetry (OTel) is a CNCF project providing a vendor-neutral API, SDK, and Collector for metrics, logs, and traces. It became the standard because: (1) Portability: Instrument once, export to any backend (Jaeger, Datadog, New Relic, Prometheus, etc.) by just changing configuration. Previously you’d be locked into each vendor’s SDK. (2) Single standard: Merges OpenCensus and OpenTracing into one unified standard, ending fragmentation. (3) All pillars: Covers metrics, logs, AND traces in one system. (4) Auto-instrumentation: Many frameworks are automatically instrumented (Flask, Django, Spring, etc.) with zero code changes.

---

27.9 Summary

Concept	Key Point
Trace	Complete request path across services
Span	Single unit of work (one service’s contribution)
TraceID	Links all spans of one request
Context Propagation	Headers (traceparent) pass context between services
OTel Collector	Vendor-neutral hub: receive, process, export
Tail Sampling	Sample by outcome (always keep errors)

---

Next Chapter: Chapter 28: SLOs, SLAs & Error Budgets

Prerequisites

Chapter 28 (Metrics, Logs, Traces).

---

Advantages & Disadvantages

Advantages: Vendor-neutral standard, instrument code once and send to any backend. Disadvantages: Still evolving rapidly, auto-instrumentation can sometimes add latency.

---

Common Mistakes

• Manual instrumentation instead of using auto-instrumentation libraries where available.
• Not sampling traces at the edge (gateway) — tracing 100% of traffic in a high-volume system will overwhelm the backend.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Traces missing spans (broken traces)	Context propagation failure between services	Check HTTP headers (traceparent) in logs	Ensure all services use the same context propagation format (W3C Trace Context)
Collector dropping spans	Backend (Jaeger) too slow or collector memory full	Check collector metrics: otelcol_processor_dropped_spans	Scale the backend, increase collector queue size, or adjust sampling rate

---

Hands-on Lab

Objective: Trace a request using OpenTelemetry.

# 1. Run a local Jaeger instance
docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

# 2. Check the OpenTelemetry Collector config
cat /etc/otelcol/config.yaml

---

Exercises

1. Instrument a simple Python Flask application using the OpenTelemetry auto-instrumentation wrapper and send traces to a local Jaeger instance.
2. Configure the OpenTelemetry collector to use tail-based sampling, keeping 100% of traces with errors, and 5% of successful traces.

---

Revision Notes

• OpenTelemetry (OTel) is the standard for generating and collecting telemetry data.
• OTel uses the W3C Trace Context standard for passing trace IDs between services.
• The OTel Collector decouples instrumentation from the storage backend (Jaeger, Zipkin, Datadog).

---

Further Reading

• OpenTelemetry Documentation
• W3C Trace Context Specification

---

Related Chapters

• Chapter 28 — Observability fundamentals

---

Last Updated: July 2026