Keepalived_haproxy
Chapter 72: High Availability and Load Balancing
Section titled “Chapter 72: High Availability and Load Balancing”Overview
Section titled “Overview”High Availability (HA) and Load Balancing are critical for production environments. This chapter covers HA concepts, keepalived, HAProxy, load balancing strategies, and building resilient infrastructure.
72.1 High Availability Concepts
Section titled “72.1 High Availability Concepts”HA Fundamentals
Section titled “HA Fundamentals” High Availability Architecture+------------------------------------------------------------------+| || Single Point of Failure High Availability || +----------+ +----------+ || | Server | | LB | || | | | | | | || | v | | +---+---+---+ || | App | | | | | | || +----------+ | v v v v || +--+---+---+---+ || |S1 S2 S3 | || +--+---+---+---+ || Server Cluster |+------------------------------------------------------------------+Key HA Concepts
Section titled “Key HA Concepts”# SLA (Service Level Agreement)# 99.9% = 8.76 hours downtime/year# 99.99% = 52.6 minutes downtime/year# 99.999% = 5.26 minutes downtime/year
# HA Architecture Components# - Redundancy: Multiple instances of everything# - Failover: Automatic switching on failure# - Load balancing: Distribute traffic# - Monitoring: Detect failures quickly# - Health checks: Verify service availabilityHA Designs
Section titled “HA Designs”# Active-Passive# - Primary server handles traffic# - Secondary server waits# - Failover on primary failure
# Active-Active# - All servers handle traffic# - Better resource utilization# - More complex setup
# N+1 Redundancy# - N servers needed for load# - 1 extra server for failover
# Geographic Redundancy# - Multiple data centers# - DNS failover# - Data replication72.2 keepalived
Section titled “72.2 keepalived”Installing keepalived
Section titled “Installing keepalived”# Install keepalivedsudo pacman -S keepalived
# Enable and startsudo systemctl enable --now keepalivedkeepalived Configuration
Section titled “keepalived Configuration”# VRRP for IP failover
vrrp_instance VI_1 { state MASTER # BACKUP on other servers interface eth0 virtual_router_id 51
priority 100 # 100 on master, 90 on backup advert_int 1
authentication { auth_type PASS auth_pass secret123 }
virtual_ipaddress { 192.168.1.100/24 dev eth0 }
# Notification scripts notify_master /etc/keepalived/notify.sh master notify_backup /etc/keepalived/notify.sh backup notify_fault /etc/keepalived/notify.sh fault}Script for Health Checks
Section titled “Script for Health Checks”#!/bin/bashcase "$1" in master) echo "Became MASTER" | logger # Start services systemctl start nginx ;; backup) echo "Became BACKUP" | logger ;; fault) echo "FAULT state" | logger ;;esacKeepalived with Health Checks
Section titled “Keepalived with Health Checks”vrrp_script check_nginx { script "/usr/bin/pgrep nginx" interval 2 timeout 2 fall 3 rise 2}
vrrp_instance VI_1 { # ... other config ...
track_script { check_nginx }}72.3 HAProxy
Section titled “72.3 HAProxy”Installing HAProxy
Section titled “Installing HAProxy”# Install HAProxysudo pacman -S haproxy
# Start servicesudo systemctl enable --now haproxyBasic HAProxy Configuration
Section titled “Basic HAProxy Configuration”global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon
# Default SSL settings ssl-default-bind-ciphers PROFILE=SYSTEM ssl-default-server-ciphers PROFILE=SYSTEM
defaults log global mode http option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http
frontend http_front bind *:80 bind *:443 ssl crt /etc/ssl/certs/server.pem
default_backend web_servers
backend web_servers balance roundrobin option httpchk GET /health
server web1 192.168.1.10:80 check inter 2000 rise 2 fall 3 server web2 192.168.1.11:80 check inter 2000 rise 2 fall 3 server web3 192.168.1.12:80 check inter 2000 rise 2 fall 3HAProxy Load Balancing Algorithms
Section titled “HAProxy Load Balancing Algorithms”# Round Robin (default)balance roundrobin
# Least Connectionsbalance leastconn
# Source IP (persistence)balance source
# URIbalance uri
# URL parameterbalance url_param
# Headerbalance hdr(User-Agent)HAProxy Health Checks
Section titled “HAProxy Health Checks”# Basic TCP checkoption tcpchk
# HTTP checkoption httpchkhttp-check expect status 200
# Custom health checkhttp-check GET /api/health HTTP/1.1\r\nHost:\ example.com
# Enable statslisten stats bind *:8404 stats enable stats uri /stats stats refresh 30s stats auth admin:passwordSSL Termination with HAProxy
Section titled “SSL Termination with HAProxy”frontend https_front bind *:443 ssl crt /etc/ssl/certs/server.pem crt /etc/ssl/certs/
# Redirect HTTP to HTTPS http-request redirect scheme https unless { ssl_fc }
default_backend web_servers
# Backend with SSLbackend web_servers balance roundrobin option ssl-hello-chk server web1 192.168.1.10:443 check ssl verify none server web2 192.168.1.11:443 check ssl verify none72.4 nginx as Load Balancer
Section titled “72.4 nginx as Load Balancer”nginx Configuration
Section titled “nginx Configuration”http { upstream backend { least_conn;
server 192.168.1.10:80 weight=3; server 192.168.1.11:80; server 192.168.1.12:80 backup;
keepalive 32; }
server { listen 80;
location / { proxy_pass http://backend; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; }
# Health check endpoint location /health { access_log off; return 200 "healthy\n"; add_header Content-Type text/plain; } }}nginx with SSL Termination
Section titled “nginx with SSL Termination”# SSL configurationserver { listen 443 ssl http2;
ssl_certificate /etc/ssl/certs/server.crt; ssl_certificate_key /etc/ssl/certs/server.key;
ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m;
location / { proxy_pass http://backend; }}72.5 DNS Load Balancing
Section titled “72.5 DNS Load Balancing”Round Robin DNS
Section titled “Round Robin DNS”zone "example.com" { type master; file "db.example.com";};
# Zone file@ IN A 192.168.1.10@ IN A 192.168.1.11@ IN A 192.168.1.12www IN A 192.168.1.10www IN A 192.168.1.11www IN A 192.168.1.12Cloud DNS (Route 53)
Section titled “Cloud DNS (Route 53)”# AWS CLI examplesaws route53 create-hosted-zone --name example.com --caller-reference "unique-id"
# Create record set with health checkaws route53 change-resource-record-sets \ --hosted-zone-id Z1234567890 \ --change-batch '{ "Changes": [{ "Action": "CREATE", "ResourceRecordSet": { "Name": "example.com", "Type": "A", "SetIdentifier": "primary", "HealthCheckId": "abc123", "AliasTarget": { "HostedZoneId": "Z2FDTNDATAQYW2", "DNSName": "dualstack.elb-123456789.us-east-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } }] }'72.6 High Availability Patterns
Section titled “72.6 High Availability Patterns”Database HA
Section titled “Database HA” Database HA Pattern+------------------------------------------------------------------+| || Primary DB Standby DB || +----------+ +----------+ || | |----WAL------>| | || | Primary | Shipping | Standby | || | | | | || +----------+ +----------+ || |+------------------------------------------------------------------+PostgreSQL HA with Patroni
Section titled “PostgreSQL HA with Patroni”scope: postgresname: postgresql0
restapi: listen: 127.0.0.1:8008 connect_address: 127.0.0.1:8008
postgresql: listen: 127.0.0.1:5432 data_dir: /data/postgresql0 pgpass: /tmp/pgpass authentication: replication: username: replicator password: password
etcd: hosts: 127.0.0.1:2379Redis Sentinel
Section titled “Redis Sentinel”# sentinel.confsentinel monitor mymaster 127.0.0.1 6379 2sentinel down-after-milliseconds mymaster 5000sentinel failover-timeout mymaster 60000sentinel parallel-syncs mymaster 172.7 Monitoring HA Setup
Section titled “72.7 Monitoring HA Setup”Health Check Script
Section titled “Health Check Script”#!/bin/bash# Check if VIP is assignedvip=$(ip addr show | grep 192.168.1.100)if [ -z "$vip" ]; then echo "CRITICAL: VIP not assigned" exit 2fi
# Check backend serversbackend_status=$(curl -s http://192.168.1.10:80/health)if [ "$backend_status" != "healthy" ]; then echo "WARNING: Backend 1 unhealthy"fi
# Check haproxyhaproxy_check=$(systemctl is-active haproxy)if [ "$haproxy_check" != "active" ]; then echo "CRITICAL: HAProxy not running" exit 2fi
echo "OK: HA setup healthy"exit 072.8 Complete HA Example
Section titled “72.8 Complete HA Example”Architecture Diagram
Section titled “Architecture Diagram” Complete HA Architecture+------------------------------------------------------------------+| || Client || | || v || DNS || +----+----+ || | | || v v || +----+ +----+ || | LB1| | LB2| || +----+ +----+ || | | || +---+-----+ || | || +---+-----+-----+ || | | || v v || +-------+ +-------+ || |App Srv| |App Srv| || | 1 | | 2 | || +-------+ +-------+ || | | || +------+-----+ || | || v || +----------+ || |Primary DB| || +----------+ || | || v || +----------+ || |Standby DB| || +----------+ || |+------------------------------------------------------------------+Complete HAProxy with Keepalived
Section titled “Complete HAProxy with Keepalived”# /etc/keepalived/keepalived.conf (on both lb1 and lb2)
vrrp_script haproxy_check { script "systemctl is-active haproxy" interval 2 timeout 2 fall 3 rise 2}
vrrp_instance HA_VIP { state BACKUP interface eth0 virtual_router_id 50
priority 100 # 100 on lb1, 90 on lb2 advert_int 1
authentication { auth_type PASS auth_pass haproxy_secret }
virtual_ipaddress { 192.168.1.100/24 }
track_script { haproxy_check }}Summary
Section titled “Summary”In this chapter, you learned:
- ✅ High availability concepts and architecture
- ✅ SLA and uptime calculations
- ✅ keepalived for IP failover
- ✅ HAProxy load balancing
- ✅ nginx as load balancer
- ✅ DNS load balancing
- ✅ Database HA patterns
- ✅ HA monitoring and health checks
- ✅ Complete HA architecture examples
Next Chapter
Section titled “Next Chapter”Chapter 20: Troubleshooting Methodology
Last Updated: February 2026