High Availability: HAProxy, Keepalived, VRRP & Pacemaker

Chapter 38: High Availability: HAProxy, Keepalived, VRRP & Pacemaker

Learning Objectives

• Design HA architectures with no single points of failure
• Configure HAProxy for load balancing and health checks
• Implement VIP failover with Keepalived and VRRP
• Understand cluster resource management with Pacemaker/Corosync

---

What is this? (Beginner On-Ramp)

High Availability (HA) is about eliminating single points of failure so a system never goes down. If a server dies, a network switch breaks, or a data center loses power, an HA system automatically shifts the work to backup equipment so users never notice.

38.1 High Availability Concepts

    Availability Mathematics
    ─────────────────────────

    Availability = Uptime / (Uptime + Downtime)

    Two Nines:   99%    = 87.6 hours/year downtime
    Three Nines: 99.9%  = 8.76 hours/year
    Four Nines:  99.99% = 52.6 minutes/year
    Five Nines:  99.999% = 5.26 minutes/year

    Cost of HA doubles roughly with each nine.

    Series vs Parallel Components:
    ────────────────────────────────
    Series (single path): A → B → C
    Availability = A × B × C
    If A=B=C=99%: 0.99³ = 97.03% — worse than each component!

    Parallel (redundant): A₁ or A₂ → B₁ or B₂
    Availability = 1 - ((1-A₁) × (1-A₂))
    If A₁=A₂=99%: 1 - (0.01²) = 99.99% — much better!

    → Redundancy transforms series into parallel paths

    Single Points of Failure (SPOFs) to eliminate:
    • Single load balancer
    • Single database master
    • Single network switch
    • Single power supply
    • Single data center

---

38.2 HAProxy Configuration

    HAProxy Architecture
    ─────────────────────

    Clients
       │
       ▼
    ┌─────────────────────────────┐
    │       HAProxy               │
    │  Frontend (accepts traffic) │
    │        │                    │
    │  ACLs & routing             │
    │        │                    │
    │  Backend (server group)     │
    │  health checks, balancing   │
    └──────┬──────────────────────┘
           │
    ┌──────┼──────────────┐
    ▼      ▼              ▼
  web-01  web-02       web-03

/etc/haproxy/haproxy.cfg

global
    log /dev/log local0 warning
    maxconn 50000
    user haproxy
    group haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    option  forwardfor         # Add X-Forwarded-For header
    option  http-server-close  # Enable HTTP keep-alive

# ── Stats Dashboard ───────────────────────────────────────
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:secure_password

# ── HTTP Frontend ─────────────────────────────────────────
frontend http_front
    bind *:80
    redirect scheme https code 301

frontend https_front
    bind *:443 ssl crt /etc/ssl/certs/api.example.com.pem

    # ACL-based routing
    acl is_api    hdr(host) -i api.example.com
    acl is_admin  hdr(host) -i admin.example.com

    use_backend api_servers   if is_api
    use_backend admin_servers if is_admin
    default_backend api_servers

# ── API Backend ───────────────────────────────────────────
backend api_servers
    balance leastconn           # Least connections (good for APIs)
    # balance roundrobin        # Round robin
    # balance source            # Sticky by source IP

    option httpchk GET /health HTTP/1.1\r\nHost:\ api.example.com
    http-check expect status 200

    default-server inter 5s fall 3 rise 2 on-marked-down shutdown-sessions

    server web-01 10.0.1.10:8080 check weight 1
    server web-02 10.0.1.11:8080 check weight 1
    server web-03 10.0.1.12:8080 check weight 1
    server web-04 10.0.1.13:8080 check weight 1 backup  # Backup: only if others fail

# ── TCP Backend (PostgreSQL) ──────────────────────────────
frontend postgres_front
    bind *:5432
    mode tcp
    default_backend postgres_servers

backend postgres_servers
    mode tcp
    option tcp-check

    server pg-primary 10.0.2.10:5432 check
    server pg-standby 10.0.2.11:5432 check backup  # Use standby only if primary fails

# HAProxy management
haproxy -c -f /etc/haproxy/haproxy.cfg   # Validate config
systemctl reload haproxy                   # Reload (zero downtime)

# Runtime API (via stats socket)
echo "show servers state" | socat stdio /run/haproxy/admin.sock
echo "disable server api_servers/web-03" | socat stdio /run/haproxy/admin.sock  # Drain server
echo "enable server api_servers/web-03" | socat stdio /run/haproxy/admin.sock

---

38.3 Keepalived & VRRP for VIP Failover

    VRRP Virtual IP Failover
    ──────────────────────────

    ┌─────────────────────────────────────────────────────┐
    │  VIP: 10.0.0.100 (virtual IP — clients connect here) │
    └──────────────────────────────────────────────────────┘
           │ Owned by MASTER
    ┌──────┴──────┐         ┌─────────────┐
    │  lb-01      │ VRRP    │  lb-02      │
    │  (MASTER)   │◄───────►│  (BACKUP)   │
    │  Priority:  │         │  Priority:  │
    │  100        │         │  90         │
    └─────────────┘         └─────────────┘

    lb-01 goes down → lb-02 takes VIP in ~1-2 seconds
    DNS entry → VIP → always up (one of the LBs has it)

# /etc/keepalived/keepalived.conf (on lb-01 — MASTER)

global_defs {
    router_id lb-01
    enable_script_security
}

vrrp_script check_haproxy {
    script "/usr/bin/killall -0 haproxy"  # Returns 0 if haproxy running
    interval 2
    weight -10              # Drop priority by 10 if script fails
}

vrrp_instance VI_1 {
    state MASTER            # BACKUP on the other node
    interface eth0
    virtual_router_id 51   # Same on both nodes
    priority 100           # 90 on BACKUP node
    advert_int 1           # Send VRRP ads every 1 second
    nopreempt              # Don't reclaim MASTER role after recovery

    authentication {
        auth_type PASS
        auth_pass secure_vrrp_password
    }

    virtual_ipaddress {
        10.0.0.100/24 dev eth0  # The VIP
    }

    track_script {
        check_haproxy
    }

    notify_master "/etc/keepalived/master.sh"   # Run on MASTER transition
    notify_backup "/etc/keepalived/backup.sh"
}

# Keepalived management
systemctl enable --now keepalived
ip addr show eth0 | grep 10.0.0.100   # Confirm VIP is on this node

# Test failover
systemctl stop haproxy    # lb-01 loses VIP, lb-02 takes over
ip addr show eth0 | grep 10.0.0.100  # VIP should be gone

systemctl start haproxy   # lb-01 haproxy restarts
# With nopreempt: lb-02 keeps VIP (no unnecessary failback)

---

38.4 Pacemaker & Corosync (Cluster Resource Manager)

Pacemaker is a full-featured cluster resource manager (used for databases, file systems, complex services).

# Install (RHEL/CentOS)
yum install pacemaker corosync pcs

# Setup cluster
pcs cluster auth node1 node2 node3 -u hacluster
pcs cluster setup --name mycluster node1 node2 node3
pcs cluster start --all

# Add VIP resource
pcs resource create virtual_ip ocf:heartbeat:IPaddr2 \
    ip=10.0.0.100 cidr_netmask=24 nic=eth0 \
    op monitor interval=10s

# Add HAProxy resource
pcs resource create haproxy systemd:haproxy \
    op monitor interval=10s

# Colocation constraint: VIP and HAProxy on same node
pcs constraint colocation add haproxy with virtual_ip INFINITY

# Ordering: VIP starts before HAProxy
pcs constraint order virtual_ip then haproxy

# Check cluster status
pcs status
crm_mon -1   # One-shot cluster status

# Stonith (fencing) — critical for split-brain prevention
pcs stonith create fence_vmware fence_vmware_soap \
    pcmk_host_map="node1:vm-node1;node2:vm-node2" \
    ipaddr=vcenter.example.com login=admin passwd=password

---

38.5 Interview Questions

Q1: What is a split-brain scenario and how is it prevented?

Answer: Split-brain occurs in a cluster when the communication between nodes fails, causing each node to think the other has failed and both attempt to take the active/master role. For example, in a two-node cluster: network partition → both nodes think the other is dead → both start the VIP/database → data corruption from two nodes writing simultaneously. Prevention: (1) Quorum: require a majority (>50%) of nodes to be reachable before taking action. A 2-node cluster can’t achieve quorum after partition (1/2 = not majority), so both nodes halt services. (2) STONITH (Shoot The Other Node In The Head): fencing — when a node is suspected dead, another node forcibly powers it off via IPMI/DRAC before taking resources. This guarantees at most one active node.

Q2: What is the difference between HAProxy’s roundrobin and leastconn balancing algorithms?

Answer: Roundrobin distributes requests sequentially: request 1→server1, 2→server2, 3→server3, 4→server1, etc. Simple and fair for stateless short-lived requests (HTTP, REST APIs). Works well when requests have similar processing time. Leastconn sends each new request to the server with fewest active connections. Better for long-lived connections (WebSockets, database connections, SOAP) or variable processing times where some requests take much longer. If server1 is stuck processing 50 slow requests and server2 has 5, leastconn sends to server2 while roundrobin would still alternate.

---

38.6 Summary

Component	Purpose
HAProxy	Layer 4/7 load balancer, health checks
Keepalived	VIP failover via VRRP
VRRP	Protocol for virtual IP handoff
Pacemaker	Cluster resource manager
Corosync	Cluster communication layer
Quorum	Prevents split-brain
STONITH	Fencing — guarantee one active node

---

Next Chapter: Chapter 39: Disaster Recovery: RPO, RTO, Backup Strategies

Prerequisites

Networking basics.

---

Advantages & Disadvantages

Advantages: Near 100% uptime, users never notice hardware failures. Disadvantages: Doubles or triples infrastructure costs, introduces ‘split-brain’ risks.

---

Common Mistakes

• Creating active-active setups for databases that only support active-passive, leading to split-brain.
• Relying on a single load balancer (single point of failure).
• Not configuring proper fencing/STONITH in clustering, causing data corruption during network partitions.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Split-brain in cluster	Network partition between nodes without fencing	Check cluster status (e.g., pcs status)	Implement STONITH to kill isolated nodes
Keepalived VIP not failing over	Priority misconfigured or VRRP blocked by firewall	journalctl -u keepalived; tcpdump -i eth0 vrrp	Fix priorities; allow VRRP multicast traffic

---

Hands-on Lab

Objective: Inspect a HAProxy configuration.

# 1. View HAProxy config (if installed)
# cat /etc/haproxy/haproxy.cfg

# 2. Check HAProxy stats socket
# echo "show info" | socat stdio /var/run/haproxy.stat

---

Exercises

1. Set up two VMs with Keepalived to share a Virtual IP (VIP). Stop Keepalived on the master and verify the VIP moves to the backup.
2. Configure HAProxy to load balance traffic between two backend web servers using round-robin.

---

Revision Notes

• HA eliminates Single Points of Failure (SPOFs).
• Keepalived provides VIP failover using the VRRP protocol.
• Fencing (STONITH - Shoot The Other Node In The Head) is critical to prevent split-brain in stateful clusters.
• Active-Active scales reads; Active-Passive ensures consistency for writes.

---

Further Reading

• HAProxy Documentation
• Keepalived Documentation

---

Related Chapters

• Chapter 40 — Load Balancing

---

Last Updated: July 2026