Load Balancing Strategies & Algorithms

Chapter 40: Load Balancing Strategies & Algorithms

Learning Objectives

Understand Layer 4 vs Layer 7 load balancing differences
Choose the right load balancing algorithm for each workload
Implement session persistence and health checking
Understand global load balancing and anycast routing

What is this? (Beginner On-Ramp)

Load balancing is like a traffic cop for the internet. When thousands of users visit a website, a load balancer sits in front and distributes those requests evenly across dozens of backend servers so that no single server gets overwhelmed and crashes.

40.1 Layer 4 vs Layer 7 Load Balancing

    Layer 4 (Transport Layer)
    ──────────────────────────
    Works at TCP/UDP level — sees IP addresses and ports
    Cannot read HTTP headers, paths, or content

    Client ──────────────────────────────► Server
    (TCP SYN to VIP)  LB forwards TCP flow  (Direct connection)

    ✓ Very fast (packet-level forwarding, ~10M+ connections/s)
    ✓ Works for any protocol (MySQL, Redis, gRPC, HTTP)
    ✓ Lower latency (no SSL termination at LB by default)
    ✗ No content-based routing
    ✗ Cannot inspect or modify HTTP headers

    Tools: HAProxy (tcp mode), NGINX stream, AWS NLB, Linux IPVS

    Layer 7 (Application Layer)
    ────────────────────────────
    Terminates TCP connection, reads HTTP/HTTPS content
    Routes based on URL, headers, cookies, method

    Client ──► LB ──► Server
    (Two TCP connections: client↔LB, LB↔server)

    ✓ Content-based routing (path, header, host)
    ✓ SSL termination (offload crypto from servers)
    ✓ Request/response modification (add headers, rewrite URLs)
    ✓ Session affinity via cookie
    ✓ Response caching
    ✗ Slower than L4 (more CPU, terminates connections)

    Tools: HAProxy (http mode), NGINX, Traefik, AWS ALB, Envoy

40.2 Load Balancing Algorithms

    Algorithm Comparison
    ─────────────────────

    ROUND ROBIN:
    Requests: 1→S1, 2→S2, 3→S3, 4→S1, 5→S2...
    Best for: Stateless, similar request cost, homogeneous servers

    WEIGHTED ROUND ROBIN:
    S1 weight=3, S2 weight=1 → S1 gets 3x traffic
    Best for: Heterogeneous servers (different capacities)
    HAProxy: server web-01 10.0.1.10 weight 3

    LEAST CONNECTIONS:
    New request → server with fewest active connections
    Best for: Long-lived connections (WebSocket, DB, gRPC streaming)
    HAProxy: balance leastconn

    LEAST RESPONSE TIME (FASTEST):
    New request → server with lowest latency
    Best for: Latency-sensitive where server performance varies
    HAProxy: balance first (+ health check latency)

    IP HASH (Source Hash):
    Client IP → always same server (hash(IP) % N)
    Best for: Simple sticky sessions without cookies
    NGINX: ip_hash;
    Downside: Uneven distribution (some IPs more frequent)

    CONSISTENT HASH:
    Hash(request_key) → server on ring
    Adding/removing servers minimizes remapping
    Best for: Caches (same request hits same cache node)

    RANDOM (with 2 choices — Power of Two Choices):
    Pick 2 servers randomly, send to less loaded one
    Better than random, simpler than leastconn
    Used by: Nginx, Envoy, Linkerd

40.3 Health Checking

    Health Check Types
    ───────────────────

    1. TCP Check (basic): Can I connect?
       TCP SYN → SYN-ACK = healthy
       Fast but doesn't test application logic

    2. HTTP Check (better): Does it return 200?
       GET /health → 200 OK = healthy
       Tests HTTP stack is working

    3. Deep Health Check (best): Is it actually working?
       GET /health/deep → checks:
       - Database connectivity
       - Cache connectivity
       - Disk space
       - Memory
       Returns 200 only if ALL dependencies are healthy
       → Don't send traffic to a server that can't serve users!

# HAProxy health check config
backend api_servers
    # HTTP health check
    option httpchk GET /health HTTP/1.1\r\nHost:\ api.example.com
    http-check expect status 200

    # Check interval and thresholds:
    # inter 5s: check every 5 seconds
    # fall 3: mark DOWN after 3 consecutive failures
    # rise 2: mark UP after 2 consecutive successes
    default-server inter 5s fall 3 rise 2

    server web-01 10.0.1.10:8080 check
    server web-02 10.0.1.11:8080 check

# NGINX upstream health check (nginx-plus or ngx_http_upstream_hc_module)
upstream backend {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;

    # Passive health check (built-in)
    # max_fails=3: mark down after 3 consecutive failures
    # fail_timeout=30s: time window + how long to stay down
}

40.4 Session Persistence (Sticky Sessions)

    Why Sticky Sessions Are Needed:
    ─────────────────────────────────
    User logs in → session stored on server-01
    Next request → routed to server-02 → "not logged in!"

    Solutions:
    ──────────

    Best: Centralized session storage
    → Redis/Memcached stores sessions (not on servers)
    → Any server can serve any user
    → No need for sticky sessions!

    When you must use sticky: Legacy apps with server-side sessions

# HAProxy: Cookie-based sticky sessions
backend api_servers
    cookie SERVERID insert indirect nocache  # LB inserts cookie

    server web-01 10.0.1.10:8080 check cookie web-01
    server web-02 10.0.1.11:8080 check cookie web-02

# Client gets: Set-Cookie: SERVERID=web-01
# Next request → LB reads cookie → sends to web-01

# Problem: If web-01 goes down, all its sessions are lost
# → Design stateless apps + Redis sessions instead!

40.5 Global Load Balancing

    Global Traffic Management
    ──────────────────────────

    Option 1: GeoDNS
    DNS returns different IPs based on client location
    client → DNS → IP based on GeoIP lookup → nearest region

    ✓ Simple, no extra infrastructure
    ✗ Slow failover (DNS TTL, resolver caching)
    ✗ Not truly global (DNS is per-query, not per-packet)

    Option 2: Anycast
    Same IP announced from multiple locations via BGP
    Client packet → routed to nearest datacenter automatically

    ✓ Fast failover (BGP convergence ~minutes)
    ✓ DDoS mitigation (traffic absorbed across many PoPs)
    ✓ Used by: Cloudflare, Google, Fastly, AWS Global Accelerator
    ✗ Requires BGP and IP ownership

    How Anycast Works:
    ───────────────────
    All PoPs announce 1.2.3.4/32 via BGP
    Internet routes to NEAREST PoP (shortest AS path)
    PoP A goes down → BGP withdrawal → traffic reroutes to PoP B

    Option 3: Service Mesh (Envoy/Istio Global LB)
    Control plane aware of all regions
    Smart routing: latency-based, failover, traffic splitting
    ✓ Most sophisticated, request-level decisions
    ✗ Complex

40.6 Modern Service Mesh Load Balancing (Envoy)

# Envoy: Advanced LB features used by Istio, Linkerd, Consul Connect

# Virtual cluster with weighted routing (canary deployment)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: checkout-service
spec:
  hosts:
    - checkout-service
  http:
  - match:
    - headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: checkout-service
        subset: canary
      weight: 100

  - route:  # Default: 95% stable, 5% canary
    - destination:
        host: checkout-service
        subset: stable
      weight: 95
    - destination:
        host: checkout-service
        subset: canary
      weight: 5

  # Retry policy
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx,gateway-error,connect-failure

  # Circuit breaker (in DestinationRule)
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: checkout-service
spec:
  host: checkout-service
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50    # Eject at most 50% of instances
  subsets:
  - name: stable
    labels:
      version: stable
  - name: canary
    labels:
      version: canary

40.7 Interview Questions

Q1: When should you use Layer 4 vs Layer 7 load balancing?

Answer: Use Layer 4 when: (1) Performance is critical — L4 is significantly faster since it just forwards TCP packets, not terminating connections. (2) Protocol-agnostic — database connections (MySQL, PostgreSQL), Redis, gRPC without HTTP. (3) SSL passthrough — you want end-to-end encryption without terminating at LB. Use Layer 7 when: (1) Content-based routing — different paths to different services (api.example.com/orders → order service, /payment → payment service). (2) SSL termination — offload crypto from application servers. (3) Request modification — add headers, A/B test routing, canary deployments. (4) WebSockets — need to manage upgrade protocol. Most production setups use both: L4 (AWS NLB) for the external entry, L7 (ALB/nginx/haproxy) for application routing.

40.8 Summary

Algorithm	Best For
Round Robin	Stateless, homogeneous servers
Weighted RR	Different server capacities
Least Connections	Long-lived connections
IP Hash	Basic sticky sessions
Consistent Hash	Cache servers
Power of 2 Choices	Large clusters
Avoid sticky sessions	Use Redis instead

Next Chapter: Chapter 41: SRE Principles: Error Budgets, Reliability & Toil

Prerequisites

Chapter 38 (High Availability).

Advantages & Disadvantages

Advantages: Scales traffic infinitely, automatically removes dead servers from rotation. Disadvantages: L7 load balancing is CPU-intensive, can become a bottleneck itself.

Common Mistakes

Using strict round-robin when backend servers have different capacities.
Not configuring health checks, leading the LB to send traffic to dead nodes.
Terminating TLS at the LB without securing the backend connection in zero-trust environments.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Uneven traffic distribution	Sticky sessions enabled or long-lived Keep-Alive connections	`Check LB stats and backend connection counts`	Use Least Connections algorithm; tune Keep-Alive timeouts
502 Bad Gateway errors	Backend servers down or rejecting connections	`Check backend logs; verify health check configuration`	Fix backend service; ensure LB health checks accurately reflect service state

Hands-on Lab

Objective: Observe load balancing behavior.

# 1. Use curl to hit a load balanced endpoint (e.g., multiple times)
# for i in {1..10}; do curl -s http://lb-address | grep "Server"; done

Exercises

Configure Nginx as a reverse proxy/load balancer with three backends. Implement a health check that requires a 200 OK response from /health.
Change the load balancing algorithm from Round Robin to IP Hash and observe the routing behavior using curl.

Revision Notes

Layer 4 (Transport) routing is fast but blind to content. Layer 7 (Application) can route based on HTTP headers/paths.
Algorithms: Round Robin, Least Connections, IP Hash.
Health checks are critical: passive (observing traffic) vs active (polling endpoints).
Sticky sessions (Session Affinity) tie a user to a specific backend, often needed for legacy apps.