DPDK, XDP & Kernel Bypass Networking

Chapter 49: DPDK, XDP & Kernel Bypass Networking

Learning Objectives

• Understand when kernel bypass networking is needed
• Build a DPDK application for high-performance packet processing
• Compare DPDK vs XDP vs SR-IOV approaches
• Understand use cases in telecom, HFT, and cloud networking

Note: XDP architecture and comparison was introduced in Chapter 27. This chapter goes deeper into DPDK application development and SR-IOV.

---

What is this? (Beginner On-Ramp)

DPDK and XDP are technologies for extreme network speed. Normal networking is too slow for things like stock trading or massive telecom routers. These tools bypass standard bottlenecks, allowing servers to process tens of millions of network packets per second.

49.1 DPDK Architecture (Deep Dive)

    DPDK Memory Model
    ──────────────────

    Huge Pages (avoid TLB pressure)
    ────────────────────────────────
    Normal page: 4KB → many TLB entries needed for 1GB of NIC buffers
    Huge page: 2MB or 1GB → drastically fewer TLB entries

    DPDK requires: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

    Memory Pools (rte_mempool)
    ─────────────────────────────
    Pre-allocated: all packet buffers allocated at startup
    No malloc/free during packet processing
    Packets from pool: no heap allocation overhead

    Poll Mode Drivers (PMDs)
    ─────────────────────────
    NIC → DMA → Huge page memory
    CPU polls NIC ring (busy-wait) → no IRQ overhead
    Same CPU core: poll → process → transmit → repeat

    Core Pinning
    ─────────────
    dpdk_app --lcores 2,3,4   # Pins DPDK to cores 2,3,4
    Remaining cores for OS: 0,1
    isolcpus=2,3,4 → kernel never schedules tasks on those cores

---

49.2 DPDK Application Development

// dpdk_l2fwd.c — Simple L2 packet forwarder with DPDK

#include <rte_eal.h>
#include <rte_ethdev.h>
#include <rte_mbuf.h>
#include <rte_launch.h>

#define RX_RING_SIZE 1024
#define TX_RING_SIZE 1024
#define BURST_SIZE 32
#define NUM_MBUFS 8191
#define MBUF_CACHE_SIZE 250

static struct rte_mempool *mbuf_pool;

// Port configuration
static const struct rte_eth_conf port_conf = {
    .rxmode = { .max_lro_pkt_size = RTE_ETHER_MAX_LEN },
};

// Initialize a DPDK port
static void port_init(uint16_t port)
{
    struct rte_eth_dev_info dev_info;

    rte_eth_dev_info_get(port, &dev_info);

    // Configure the port
    rte_eth_dev_configure(port, 1, 1, &port_conf);

    // Setup RX queue
    rte_eth_rx_queue_setup(port, 0, RX_RING_SIZE,
        rte_eth_dev_socket_id(port), NULL, mbuf_pool);

    // Setup TX queue
    rte_eth_tx_queue_setup(port, 0, TX_RING_SIZE,
        rte_eth_dev_socket_id(port), NULL);

    // Start the port
    rte_eth_dev_start(port);
    rte_eth_promiscuous_enable(port);
}

// Main packet processing loop (runs on dedicated CPU core)
static void lcore_main(void)
{
    uint16_t port;
    struct rte_mbuf *bufs[BURST_SIZE];

    printf("Core %u forwarding packets\n", rte_lcore_id());

    // Busy poll loop — never sleeps!
    while (1) {
        // Try each port
        RTE_ETH_FOREACH_DEV(port) {
            // Receive burst of packets
            uint16_t nb_rx = rte_eth_rx_burst(port, 0, bufs, BURST_SIZE);
            if (nb_rx == 0) continue;

            // Process each packet
            for (int i = 0; i < nb_rx; i++) {
                struct rte_ether_hdr *eth_hdr = rte_pktmbuf_mtod(bufs[i], struct rte_ether_hdr *);

                // Example: count packets by EtherType
                uint16_t ether_type = rte_be_to_cpu_16(eth_hdr->ether_type);
                // ... processing ...
            }

            // Forward to other port (simple 0↔1 forwarding)
            uint16_t dst_port = port ^ 1;
            uint16_t nb_tx = rte_eth_tx_burst(dst_port, 0, bufs, nb_rx);

            // Free unsent packets
            for (int i = nb_tx; i < nb_rx; i++)
                rte_pktmbuf_free(bufs[i]);
        }
    }
}

int main(int argc, char *argv[])
{
    // Initialize DPDK Environment Abstraction Layer
    int ret = rte_eal_init(argc, argv);
    if (ret < 0) rte_exit(EXIT_FAILURE, "EAL init failed\n");

    // Create memory pool for packet buffers
    mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL",
        NUM_MBUFS, MBUF_CACHE_SIZE, 0,
        RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());

    // Initialize all ports
    uint16_t port;
    RTE_ETH_FOREACH_DEV(port) {
        port_init(port);
    }

    // Launch packet processing on main core
    lcore_main();

    return 0;
}

---

49.3 DPDK Environment Setup

# ── Install DPDK ──────────────────────────────────────────
apt install dpdk dpdk-dev dpdk-igb-uio-dkms

# ── Huge Pages Setup ──────────────────────────────────────
# 1GB huge pages (for large memory pools)
echo 4 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

# 2MB huge pages
echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Mount hugepages
mkdir -p /dev/hugepages
mount -t hugetlbfs nodev /dev/hugepages

# Persist in /etc/fstab:
# nodev /dev/hugepages hugetlbfs defaults 0 0

# ── NIC Binding ───────────────────────────────────────────
# Find NICs
dpdk-devbind.py --status-dev net

# Load UIO or VFIO driver
modprobe uio
modprobe igb_uio       # Intel UIO driver
# OR (more secure):
modprobe vfio-pci      # IOMMU-based (preferred)

# Bind NIC to DPDK driver (takes it away from kernel!)
dpdk-devbind.py --bind=igb_uio 0000:04:00.0

# Verify bound
dpdk-devbind.py --status

# ── Kernel Boot Parameters ────────────────────────────────
# /etc/default/grub:
# GRUB_CMDLINE_LINUX="hugepagesz=1G hugepages=4 iommu=on iommu_groups=on isolcpus=2,3,4,5 nohz_full=2,3,4,5 rcu_nocbs=2,3,4,5"

# ── Run DPDK Application ──────────────────────────────────
./dpdk_l2fwd \
    --lcores 0-1@0,2-3@1  \  # Core mapping
    -n 4 \                    # Memory channels
    -- -p 0x3 -T 10          # App-specific args

---

49.4 SR-IOV: Hardware Virtualization for Networking

    SR-IOV (Single Root I/O Virtualization)
    ─────────────────────────────────────────

    Without SR-IOV:
    NIC → Hypervisor (software switch) → VM
    Hypervisor must copy every packet → latency + CPU

    With SR-IOV:
    Physical NIC creates Virtual Functions (VFs)
    Each VF looks like a physical NIC to the VM/container
    VM → VF (direct hardware path) → Network
    No hypervisor packet path!

    Architecture:
    ─────────────
    Physical Function (PF): The real NIC (full control)
    Virtual Functions (VFs): Lightweight NIC clones (1 per VM/container)

    ┌─────────────────────────────────────────────────────┐
    │  Physical NIC (PF) — Intel X710, Mellanox ConnectX  │
    │                                                      │
    │  VF0  VF1  VF2  VF3  VF4  VF5  VF6  VF7  ...      │
    └──┬────┬────┬────┬────────────────────────────────────┘
       │    │    │    │
       ▼    ▼    ▼    ▼
      VM0  VM1  VM2  Container
      (Direct hardware path, no CPU copy!)

    Benefits:
    • Near bare-metal NIC performance in VMs
    • Latency: <1μs (vs 5-20μs for software switch)
    • CPU: zero CPU for packet forwarding

    Used by: cloud providers (each VM gets VF), HPC, HFT

# Enable SR-IOV on Intel X710
echo 8 > /sys/class/net/eth0/device/sriov_numvfs   # Create 8 VFs

# List VFs created
lspci | grep "Virtual Function"

# Assign VF to VM (KVM)
virsh nodedev-dumpxml pci_0000_04_02_0   # Get VF details
virsh attach-device vm1 vf-device.xml    # Attach to VM

# Kubernetes SR-IOV (SRIOV Network Device Plugin)
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/sriov-network-device-plugin/master/deployments/k8s-v1.16/sriovdp-daemonset.yaml

# Pod requesting SR-IOV interface
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: dpdk-app
    resources:
      limits:
        intel.com/intel_sriov_netdevice: "1"   # Request 1 VF

---

49.5 Interview Questions

Q1: Why does DPDK require huge pages?

Answer: DPDK processes millions of packets per second, each needing buffer memory. With standard 4KB pages, the TLB (Translation Lookaside Buffer) can’t cache all the page translations for large NIC ring buffers (often tens of MB). TLB misses force expensive page table walks on every memory access, killing performance. With 2MB huge pages, the TLB covers 512x more memory per entry — a 256MB NIC buffer pool needs 128 TLB entries with huge pages vs 65,536 with normal pages. This is why DPDK applications see dramatically better performance with huge pages: fewer TLB misses, better cache behavior for the packet buffer pool.

Q2: What is SR-IOV and how does it differ from DPDK?

Answer: They solve different problems. DPDK is a user-space library that bypasses the kernel network stack for packet processing — you write the packet processing logic. SR-IOV is a hardware feature that allows a physical NIC to present multiple virtual NIC interfaces (Virtual Functions), each appearing as a real NIC to VMs or containers. The key difference: SR-IOV gives VMs direct hardware NIC access (removing the hypervisor from the packet path), while DPDK gives user-space applications direct NIC access (removing the kernel from the packet path). They’re often combined: cloud providers use SR-IOV to give VMs near-native NIC performance, and within those VMs, network appliances might use DPDK for further packet processing optimization.

---

49.6 Summary

Technology	Bypasses	Performance	Use Case
Standard kernel	Nothing	~2 Mpps	General purpose
XDP	sk_buff alloc	~26 Mpps	Fast filtering, LB
DPDK	Entire kernel	100+ Mpps	Packet processing apps
SR-IOV	Hypervisor switch	Near bare-metal	VM/container NIC

---

Next Chapter: Chapter 50: Advanced Storage: LVM, ZFS, Ceph & NVMe

Prerequisites

Chapter 27 (eBPF & XDP), Networking fundamentals.

---

Advantages & Disadvantages

Advantages: Bypasses kernel network stack for millions of packets/sec (Telco/HFT grade). Disadvantages: DPDK burns 100% of a CPU core polling; extremely complex to write apps for.

---

Common Mistakes

• Using DPDK when XDP is sufficient — DPDK consumes 100% of a CPU core by design (busy polling); XDP uses interrupts.
• Running DPDK applications without configuring huge pages — DPDK relies on huge pages to minimize TLB misses.
• Using generic XDP instead of Native XDP and wondering why performance is poor.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
DPDK application fails to initialize EAL	Huge pages not configured or insufficient permissions	Check EAL initialization logs	Allocate huge pages (echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
XDP program doesn’t load on NIC	NIC driver doesn’t support Native XDP	ip link set dev eth0 xdp obj my_prog.o (fails)	Load with ‘xdpgeneric’ (slower) or update NIC/driver

---

Hands-on Lab

Objective: Check XDP driver support.

# 1. Check if current NIC supports native XDP
ethtool -i eth0 | grep driver

# 2. Load a dummy XDP program (if available)
# ip link set dev eth0 xdp obj xdp_drop.o sec .text

---

Exercises

1. Compare the architecture of DPDK (Kernel Bypass) vs XDP (In-Kernel Fast Path). What are the trade-offs regarding CPU utilization?
2. Write a conceptual XDP program (in C) that drops all UDP packets to port 53 (DNS) and passes everything else.

---

Revision Notes

• DPDK bypasses the kernel network stack completely, mapping NIC memory directly to user space (busy polling).
• XDP runs inside the kernel network driver, before the SKB is allocated, allowing fast drops or redirects.
• DPDK burns 100% CPU; XDP only consumes CPU when packets arrive.
• AF_XDP bridges the gap, allowing high-performance packet delivery to user space without full kernel bypass.

---

Further Reading

• DPDK Documentation
• XDP Tutorial (GitHub: xdp-project/xdp-tutorial)

---

Related Chapters

• Chapter 27 — XDP Basics

---

Last Updated: July 2026