Network Performance Analysis & Tuning

Chapter 25: Network Performance Analysis & Tuning

Learning Objectives

Diagnose network performance problems systematically
Understand TCP throughput and latency trade-offs
Tune for high-throughput, low-latency, and mixed workloads
Use modern tools: ss, nstat, nicstat, tc

What is this? (Beginner On-Ramp)

Network performance analysis is diagnosing slowness in data transfer. It’s about figuring out if the network itself is slow, if packets are getting lost along the way, or if the server is just too busy to process the data fast enough.

25.1 Network Performance Fundamentals

    Network Performance Dimensions
    ────────────────────────────────

    Throughput:   How much data per second? (MB/s, Gbps)
    Latency:      How long for one packet? (ms, μs)
    Jitter:       Variation in latency (ms variance)
    Packet Loss:  % of packets that don't arrive
    Connections:  How many concurrent TCP connections?

    The Throughput-Latency Trade-off:
    ───────────────────────────────────
    Optimizing for throughput:
    → Large buffers, batching, Nagle's algorithm
    → High latency, high throughput
    → Good for: file transfers, streaming, batch processing

    Optimizing for latency:
    → Small buffers, disable Nagle, TCP_NODELAY
    → Low latency, lower throughput
    → Good for: real-time, gaming, trading, interactive APIs

25.2 Network Diagnostics

# ── ss: Socket Statistics (replaces netstat) ─────────────
ss -s                        # Summary: TCP states, connections
ss -tnp                      # TCP connections with process info
ss -tnp state established    # Only established
ss -tnp state time-wait | wc -l   # Count TIME_WAIT

# Extended info (latency, etc.)
ss -i dst 10.0.1.5           # Connections to specific host

# ── Network Throughput ────────────────────────────────────
sar -n DEV 1 5               # Per-interface bytes/packets per second
# rxkB/s, txkB/s, rxpck/s, txpck/s

# Per-interface detailed stats
cat /proc/net/dev
ip -s link show eth0

# ── Network Errors ────────────────────────────────────────
# Interface errors:
ip -s link show eth0 | grep -A5 "RX\|TX"
# Look for: errors, dropped, overrun

# TCP errors:
nstat -a | grep -E "Retrans|Fail|Error|Reset|TimeOut"
cat /proc/net/netstat | grep -i "retran\|fail\|error"

# ── TCP Retransmissions (key health metric) ───────────────
# High retransmissions = packet loss = network congestion or bad link
watch -n 1 'nstat -z | grep RetransSegs'

# Per-connection retransmit count:
ss -i | grep retrans

# ── MTU and Fragmentation ─────────────────────────────────
ip link show eth0 | grep mtu
# Default: 1500 bytes
# With jumbo frames: 9000 bytes (requires switch support)

# Test MTU (ping with large packet):
ping -M do -s 8972 10.0.1.5   # Test for 9000-byte jumbo frames (9000 - 28 = 8972)

25.3 Latency Analysis

# ── Basic Latency ─────────────────────────────────────────
ping -c 100 10.0.1.5         # 100 pings → min/avg/max/stddev

# Better: hping3 (TCP ping, works through firewalls)
hping3 -S -p 80 -c 100 api.example.com   # TCP SYN to port 80

# MTR: traceroute + ping
mtr --report --report-cycles 100 api.example.com
# Shows per-hop latency and packet loss → find WHERE latency originates

# ── TCP Handshake Latency ─────────────────────────────────
# How long does connection establishment take?
curl -w "time_connect: %{time_connect}\ntime_ttfb: %{time_starttransfer}\ntime_total: %{time_total}\n" \
    -o /dev/null -s https://api.example.com/health

# ── Network Queuing Latency ───────────────────────────────
# Is latency happening at the NIC (hardware queue)?
tc qdisc show dev eth0
# qdisc mq 0: ...  ← Shows queuing discipline

# Monitor tx queue depth:
ip -s -h link show eth0
# TX errors (if txdrop > 0 = NIC queue full)

# ── Socket Queue Backlog ──────────────────────────────────
# Is the application draining the socket buffer fast enough?
ss -tn | head -30
# Recv-Q: data waiting to be read by application
# Send-Q: data waiting to be sent (network congestion)
# Large Recv-Q = application is slow
# Large Send-Q = network is slow

25.4 Tuning for Throughput

# Large socket buffers (for high-BDP links)
net.core.rmem_max = 134217728          # 128MB
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

# BBR congestion control (better throughput)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# Larger receive queue
net.core.netdev_max_backlog = 250000
net.core.somaxconn = 65535

# Avoid TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# Increase TCP initial window (faster small transfers)
# (typically 10 segments by default on modern kernels)

# Apply
sysctl -p /etc/sysctl.d/99-network-throughput.conf

25.5 Tuning for Low Latency

# Disable Nagle's algorithm system-wide
# (For apps: use TCP_NODELAY socket option)
# (Cannot disable globally via sysctl easily — do in application)

# TCP quick ACK
net.ipv4.tcp_quickack = 1        # (This is per-socket, set in app)

# Reduce TCP timeout
net.ipv4.tcp_fin_timeout = 10    # Faster FIN timeout

# Reduce keepalive latency
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3

# IRQ affinity (NIC interrupts to isolated CPUs)
# See Chapter 13 for details

# CPU frequency scaling: performance governor
echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

25.6 Bandwidth Testing

# iperf3: Measure actual network bandwidth between two hosts
# Server:
iperf3 -s

# Client: TCP throughput test
iperf3 -c <server_ip> -t 30 -P 8        # 8 parallel streams, 30s

# UDP test (for packet loss/jitter):
iperf3 -c <server_ip> -u -b 100M -t 30  # 100Mbps UDP

# Reverse (server→client):
iperf3 -c <server_ip> -R -t 30

# Typical results:
# 1GbE NIC: ~940 Mbps
# 10GbE NIC: ~9.4 Gbps
# 25GbE NIC: ~23+ Gbps
# If you're getting 400Mbps on 1GbE → something is wrong

25.7 Interview Questions

Q1: A service has high throughput but also high latency. What are the likely causes?

Answer: High throughput + high latency often indicates buffer bloat — network buffers are filling up (in the NIC, switch, or TCP stack) and packets wait in those buffers before being transmitted. Fixes: (1) Enable BBR congestion control, which doesn’t fill buffers the way CUBIC does. (2) Enable FQ (Fair Queuing) qdisc: net.core.default_qdisc=fq — shapes traffic to prevent buffer bloat. (3) Check for TCP retransmissions with nstat | grep Retrans — retransmits add latency. (4) Check for CPU bottleneck in network interrupt processing — if NIC queue is overflowing, enable RSS to spread across CPUs.

25.8 Summary

Metric	Tool	Target
Throughput	sar -n DEV, iperf3	< NIC max
Latency	ping, mtr	< 1ms LAN
Retransmissions	nstat RetransSegs	< 0.1%
Socket queues	ss Recv-Q	Near 0
Packet drops	ip -s link	0

Next Chapter: Chapter 26: Benchmarking, Capacity Planning & Bottleneck Analysis

Prerequisites

Chapter 11 (Network Tuning).

Advantages & Disadvantages

Advantages: Finds silent packet drops, optimizes throughput for high-traffic servers. Disadvantages: Network stacks are highly complex; debugging requires understanding TCP/IP deeply.

Common Mistakes

Using ping to test bandwidth (it only tests latency).
Assuming packet loss is a network issue — it’s often a full socket receive buffer on the host.
Running tcpdump without filters on a busy interface, causing CPU spikes and dropped packets.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
High throughput, but slow connection setup	SYN backlog full	`netstat -s	grep -i listen`
Intermittent packet loss under load	Ring buffer overflow (RX drops)	`ethtool -S eth0	grep drop`

Hands-on Lab

Objective: Benchmark and monitor network performance.

# 1. Real-time network throughput
sar -n DEV 1
# or: iftop, nload

# 2. Check socket statistics
ss -s
ss -tni  # detailed TCP internal info

# 3. Test bandwidth with iperf3 (requires server running: iperf3 -s)
# iperf3 -c server_ip -t 10

Exercises

Use iperf3 to measure TCP throughput. Then run it with -u to measure UDP throughput and observe jitter and packet loss.
Use tcpdump to capture only SYN packets (tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0') to observe connection requests.

Revision Notes

Bandwidth = capacity; Latency = delay; Throughput = actual observed transfer rate.
TCP Congestion Control (e.g., BBR, CUBIC) heavily influences throughput on high-latency networks.
ss replaces netstat for socket statistics; sar -n DEV shows interface throughput.
NIC offloads (TSO, GRO) improve performance but complicate packet capture analysis.