Load Balancing
Chapter 69: Load Balancing
Section titled “Chapter 69: Load Balancing”Comprehensive Guide to Load Balancing
Section titled “Comprehensive Guide to Load Balancing”Why This Matters in DevOps/SRE
Section titled “Why This Matters in DevOps/SRE”Load balancing is the foundation of scalable, available systems. As traffic increases, load balancers distribute requests across multiple servers, ensuring no single server becomes a bottleneck. Understanding load balancing algorithms, health checks, and session persistence is critical for building resilient infrastructure.
┌─────────────────────────────────────────────────────────────────────────────┐│ LOAD BALANCING IN SCALABLE SYSTEMS │├─────────────────────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────────────────────┐ ││ │ TRAFFIC DISTRIBUTION │ ││ │ │ ││ │ ┌──────────────┐ │ ││ │ │ Load Balancer │ │ ││ │ └───────┬───────┘ │ ││ │ │ │ ││ │ ┌────────────┼────────────┐ │ ││ │ ▼ ▼ ▼ │ ││ │ ┌────┐ ┌────┐ ┌────┐ │ ││ │ │Srv1│ │Srv2│ │Srv3│ (horizontal scaling) │ ││ │ └────┘ └────┘ └────┘ │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────────┘ ││ ││ ┌─────────────────────────────────────────────────────────────────────┐ ││ │ DEVOPS SCENARIOS │ ││ │ │ ││ │ • Traffic spikes: Auto-scale based on load │ ││ │ • Deployments: Zero-downtime with health checks │ ││ │ • Failover: Remove failed nodes automatically │ ││ │ • Geographic routing: Route to nearest datacenter │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────────┘ ││ ││ ┌─────────────────────────────────────────────────────────────────────┐ ││ │ AWS USE CASES │ ││ │ │ ││ │ ALB: HTTP/HTTPS │ NLB: TCP/UDP │ CLB: Legacy │ ││ │ Path-based routing │ High throughput│ HTTP fallback │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────────┘Real-world DevOps scenarios:
- Kubernetes: Service load balancing with kube-proxy or ingress controllers
- Blue-green deployments: Route traffic between old and new versions
- A/B testing: Route percentage of traffic to different versions
- Geographic routing: Latency-based routing with Route53
69.1 Load Balancing Algorithms
Section titled “69.1 Load Balancing Algorithms”Algorithm Comparison
Section titled “Algorithm Comparison”┌─────────────────────────────────────────────────────────────────────────┐│ ALGORITHM COMPARISON │├─────────────────────────────────────────────────────────────────────────┤│ ││ Round Robin: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Sequential distribution to each server in rotation │ ││ │ • Good for servers with similar capacity │ ││ │ • Simple, no state needed │ ││ │ • Default in most load balancers │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Least Connections: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Routes to server with fewest active connections │ ││ │ • Better for varying workloads │ ││ │ • More intelligent than round robin │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ IP Hash: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Same client IP always goes to same server │ ││ │ • Enables session persistence │ ││ │ • Good for stateful applications │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Weighted: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Assign weight to each server │ ││ │ • More requests to powerful servers │ ││ │ • Useful for heterogeneous server pools │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Least Time: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Considers response time (weighted) │ ││ │ • Routes to fastest responding servers │ ││ │ • Best for variable latency environments │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Random: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Random server selection │ ││ │ • Good when used with multiple load balancers │ ││ │ • Prevents thundering herd │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘69.2 HAProxy Configuration
Section titled “69.2 HAProxy Configuration”Basic Configuration
Section titled “Basic Configuration”global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon maxconn 4000
defaults log global mode http option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http
frontend http_front bind *:80 bind *:443 ssl crt /etc/ssl/certs/server.pem
# Redirect HTTP to HTTPS http-request redirect scheme https unless { ssl_fc }
default_backend web_servers
backend web_servers mode http balance roundrobin
option httpchk GET /health
server web1 10.0.0.1:80 check inter 2000 rise 2 fall 3 server web2 10.0.0.2:80 check inter 2000 rise 2 fall 3 server web3 10.0.0.3:80 check inter 2000 rise 2 fall 3 backupACL Examples
Section titled “ACL Examples”# Path-based routingfrontend http_front bind *:80
acl is_api path_beg /api acl is_admin path_beg /admin acl is_static path_beg /static /images /js
use_backend api_servers if is_api use_backend admin_servers if is_admin use_backend static_servers if is_static default_backend web_servers
# Header-based routingfrontend http_front bind *:80
acl is_mobile hdr_val(User-Agent) -i mobile acl is_api_header hdr(X-API-Key) -m found
use_backend mobile_servers if is_mobile use_backend api_servers if is_api_header default_backend web_serversSSL Termination
Section titled “SSL Termination”frontend https_front bind *:443 ssl crt /etc/ssl/certs/server.pem crt /etc/ssl/certs/additional.pem
# SSL options ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384
default_backend web_servers69.3 Health Checks
Section titled “69.3 Health Checks”Health Check Types
Section titled “Health Check Types”┌─────────────────────────────────────────────────────────────────────────┐│ HEALTH CHECK TYPES │├─────────────────────────────────────────────────────────────────────────┤│ ││ TCP Connect (Basic): ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Opens TCP connection to port │ ││ │ • Success if connection established │ ││ │ • server web1 10.0.0.1:80 check │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ HTTP Check: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Sends HTTP request │ ││ │ • Checks response code (2xx, 3xx = healthy) │ ││ │ • Can check response body │ ││ │ • option httpchk │ ││ │ • option httpchk GET /health │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ HTTPS Check: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • Same as HTTP but over SSL │ ││ │ • Can verify certificate │ ││ │ • option httpchk GET /health ssl verify none │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Check Parameters: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ • inter 2000: Check every 2 seconds │ ││ │ • rise 2: 2 successful checks = healthy │ ││ │ • fall 3: 3 failed checks = unhealthy │ ││ │ • slowstart 30s: Gradual traffic increase │ ││ │ • port 8080: Check different port than backend │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘Advanced Health Checks
Section titled “Advanced Health Checks”# Custom HTTP check with expected responsebackend api_servers option httpchk http-check expect status 200 http-check expect string "OK" server api1 10.0.0.1:8000 check inter 3000 fall 3 rise 2 server api2 10.0.0.2:8000 check inter 3000 fall 3 rise 2
# Check on different portbackend web_servers option httpchk http-check send uri /healthcheck port 8080 server web1 10.0.0.1:80 check inter 2000 server web2 10.0.0.2:80 check inter 2000
# Agent-based health checkbackend app_servers option external-check external-check path "/usr/bin:/bin" external-check command /usr/local/bin/check_app.sh server app1 10.0.0.1:8080 check69.4 Session Persistence
Section titled “69.4 Session Persistence”Sticky Sessions
Section titled “Sticky Sessions”# Source IP persistencebackend web_servers balance source hash-type consistent server web1 10.0.0.1:80 check server web2 10.0.0.2:80 check
# Cookie-based persistencebackend api_servers balance roundrobin cookie SERVERID insert indirect nocache server api1 10.0.0.1:80 check cookie api1 server api2 10.0.0.2:80 check cookie api269.5 Interview Questions
Section titled “69.5 Interview Questions”┌─────────────────────────────────────────────────────────────────────────┐│ LOAD BALANCING INTERVIEW QUESTIONS │├─────────────────────────────────────────────────────────────────────────┤ │Q1: What is load balancing? │ │A1: Distributing traffic across multiple servers to improve performance, │ reliability, and scalability │ │─────────────────────────────────────────────────────────────────────────┤ │Q2: What is the difference between L4 and L7 load balancing? │ │A2: L4 (Transport layer): TCP/UDP, no content awareness │ L7 (Application layer): HTTP/HTTPS, can inspect content, routes │ based on URL, headers, cookies │ │─────────────────────────────────────────────────────────────────────────┤ │Q3: What is sticky sessions and why would you use them? │ │A3: Same client always routed to same server │ - Session state stored on server │ - Enables stateful applications │ - Can use IP hash or cookies │ │─────────────────────────────────────────────────────────────────────────┤ │Q4: What health check types are available? │ │A4: TCP connect (basic), HTTP (GET request), HTTPS (SSL), │ Custom (agent-based), UDP (for non-HTTP) │ │─────────────────────────────────────────────────────────────────────────┤ │Q5: What is the difference between round robin and least connections? │ │A5: Round robin: Sequential distribution │ Least connections: Routes to server with fewest active connections│ │─────────────────────────────────────────────────────────────────────────┤ │Q6: How do you handle SSL termination? │ │A6: Load balancer decrypts SSL, forwards plain HTTP to backend │ - Reduces backend server CPU load │ - Centralized certificate management │ - HAProxy: bind *:443 ssl crt <cert> │ │─────────────────────────────────────────────────────────────────────────┤ │Q7: What is a failover in load balancing? │ │A7: Automatic switch to backup server when primary fails │ - Health checks detect failure │ - Backup servers take over │ - "backup" keyword in HAProxy │ │─────────────────────────────────────────────────────────────────────────┤ │Q8: How do you prevent a single point of failure? │ │A8: Multiple load balancers (active-active or active-passive) │ - Multiple backend servers │ - Health checks for automatic removal │ - Redundant network paths │ │└─────────────────────────────────────────────────────────────────────────┘Quick Reference
Section titled “Quick Reference”# HAProxybalance roundrobin|leastconn|source|urioption httpchkcheck inter 2000 rise 2 fall 3
# Algorithmsroundrobin # Sequentialleastconn # Fewest connectionssource # IP hashuri # URL hashCommon Mistakes & Anti-Patterns
Section titled “Common Mistakes & Anti-Patterns”1. No Health Checks
Section titled “1. No Health Checks”❌ WRONG: Traffic goes to failed servers
upstream backend { server 10.0.0.1:8080; server 10.0.0.2:8080; # No health checks!}✅ CORRECT: Add health checks
upstream backend { server 10.0.0.1:8080 max_fails=3 fail_timeout=30s; server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;}2. Wrong Algorithm for Workload
Section titled “2. Wrong Algorithm for Workload”❌ WRONG: Using round robin for long-running connections
# Bad for WebSocket or long-pollingupstream backend { server 10.0.0.1:8080; server 10.0.0.2:8080;}✅ CORRECT: Use least connections for variable workloads
upstream backend { least_conn; server 10.0.0.1:8080; server 10.0.0.2:8080;}3. Not Using Sticky Sessions When Needed
Section titled “3. Not Using Sticky Sessions When Needed”❌ WRONG: Sessions distributed across servers
# Users get different servers each requestupstream backend { server 10.0.0.1:8080; server 10.0.0.2:8080;}✅ CORRECT: Enable sticky sessions for stateful apps
upstream backend { ip_hash; # or use $cookie_jsessionid server 10.0.0.1:8080; server 10.0.0.2:8080;}4. Single Point of Failure
Section titled “4. Single Point of Failure”❌ WRONG: Single load balancer
Internet ──► LB1 ──► App Servers (if LB1 fails, everything fails)✅ CORRECT: Redundant load balancers
Internet ──► LB1 ──┬──► App Servers ──► LB2 ──┘ (Active-passive or active-active)5. Not Monitoring Load Balancer Metrics
Section titled “5. Not Monitoring Load Balancer Metrics”❌ WRONG: No visibility into LB performance
✅ CORRECT: Monitor key metrics
# Nginx stub status modulelocation /nginx_status { stub_status on; allow 127.0.0.1; deny all;}
# Key metrics to monitor:# - Requests per second# - Active connections# - Backend response times# - Health check failuresSummary
Section titled “Summary”- Algorithms: Round robin, least connections, IP hash, weighted
- HAProxy: Popular L4/L7 load balancer
- Health checks: TCP, HTTP, HTTPS, custom
- Persistence: Source IP, cookies
Next Chapter
Section titled “Next Chapter”Chapter 70: Application Servers
Last Updated: February 2026