Advanced eBPF: Writing Programs, Tracing & Observability

Chapter 48: Advanced eBPF: Writing Programs, Tracing & Observability

Learning Objectives

• Understand the eBPF program types and their use cases
• Write eBPF programs with BCC (Python) and libbpf (C)
• Use bpftrace for one-liner kernel tracing
• Build custom performance tools with eBPF

---

What is this? (Beginner On-Ramp)

Advanced eBPF involves writing complex programs in C that run directly inside the kernel’s sandbox. This allows engineers to create entirely custom performance profiling tools, security firewalls, and network routers that operate at the deepest possible level.

48.1 eBPF Architecture (Deep Dive)

    eBPF Execution Model
    ──────────────────────

    User Space          Kernel Space
    ──────────          ────────────

    Your eBPF C code
        │
        ▼
    LLVM/Clang → eBPF bytecode (.o)
        │
        ▼
    bpf() syscall → load into kernel
        │
        ▼
    eBPF Verifier (safety check):
    • No unbounded loops
    • No out-of-bounds memory access
    • All code paths reach return
    • Max 1M instructions
    • Terminates in bounded time
        │
        ▼
    JIT Compiler → native machine code
        │
        ▼
    Hook Point (triggered by event):
    ┌─────────────────────────────────────────────────────┐
    │  tracepoint:syscalls:sys_enter_openat               │
    │  kprobe:tcp_connect                                 │
    │  uprobe:/usr/bin/python3:PyObject_GetAttr           │
    │  xdp (NIC driver packet hook)                       │
    │  tc (traffic control)                               │
    │  socket filter                                      │
    │  cgroup (resource control)                          │
    └─────────────────────────────────────────────────────┘
        │
        ▼
    Maps (shared data: kernel ↔ user space)
    • Hash map, Array, Ring buffer, Stack trace map...

    User space reads results from maps

---

48.2 bpftrace: One-Liner Tracing

bpftrace is the awk/sed of eBPF — powerful one-liners for tracing.

# ── Installation ──────────────────────────────────────────
apt install bpftrace     # Ubuntu
yum install bpftrace     # RHEL/CentOS

# ── File System Tracing ───────────────────────────────────
# Trace all file opens
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s: %s\n", comm, str(args->filename)); }'

# Trace file opens by process name
bpftrace -e 'tracepoint:syscalls:sys_enter_openat /comm == "nginx"/ { printf("nginx opened: %s\n", str(args->filename)); }'

# Count file opens by process
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'

# ── Network Tracing ───────────────────────────────────────
# Trace all TCP connections
bpftrace -e 'kprobe:tcp_connect { printf("connect: %s (pid=%d)\n", comm, pid); }'

# Show DNS (port 53) requests
bpftrace -e 'tracepoint:net:net_dev_xmit /args->skbaddr->len == 53/ { printf("DNS from %s\n", comm); }'

# Count syscalls per process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:10 { print(@); }'

# ── Latency Measurement ───────────────────────────────────
# Measure read() syscall latency
bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read /@start[tid]/ {
    @latency_us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}'

# Block I/O latency histogram
bpftrace -e '
kprobe:blk_account_io_start { @start[arg0] = nsecs; }
kprobe:blk_account_io_done /@start[arg0]/ {
    @io_lat_ms = hist((nsecs - @start[arg0]) / 1000000);
    delete(@start[arg0]);
}'

# ── Process Tracing ───────────────────────────────────────
# Trace all exec() calls (new processes)
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("exec: %s -> %s\n", comm, str(args->filename)); }'

# Trace setuid (privilege escalation!)
bpftrace -e 'tracepoint:syscalls:sys_enter_setuid { printf("SETUID! pid=%d comm=%s uid=%d\n", pid, comm, args->uid); }'

# ── CPU Profiling ─────────────────────────────────────────
# CPU flamegraph (sample every 99Hz, 10 seconds)
bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' > /tmp/profile.txt 2>&1 &
sleep 10
kill %1

# ── Memory Tracing ────────────────────────────────────────
# Trace malloc calls (user space)
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { printf("malloc %d bytes from %s\n", arg0, comm); }'

---

48.3 BCC Python: Full eBPF Programs

#!/usr/bin/env python3
# tcp_connect_trace.py — Trace all outbound TCP connections

from bcc import BPF
import socket
import struct

# eBPF C program (embedded as string)
bpf_program = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

// Map to pass data to user space
BPF_PERF_OUTPUT(events);

struct event_t {
    u32 pid;
    u32 daddr;    // destination IP
    u16 dport;    // destination port
    char comm[TASK_COMM_LEN];
};

// Hook: kernel function tcp_connect
int trace_connect(struct pt_regs *ctx, struct sock *sk) {
    struct event_t event = {};

    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.daddr = sk->__sk_common.skc_daddr;
    event.dport = sk->__sk_common.skc_dport;
    bpf_get_current_comm(&event.comm, sizeof(event.comm));

    events.perf_submit(ctx, &event, sizeof(event));
    return 0;
}
"""

# Load eBPF program
b = BPF(text=bpf_program)
b.attach_kprobe(event="tcp_connect", fn_name="trace_connect")

print("Tracing TCP connections... Ctrl-C to stop")
print(f"{'PID':<8} {'COMM':<16} {'DESTINATION':<20}")

def print_event(cpu, data, size):
    event = b["events"].event(data)
    dip = socket.inet_ntoa(struct.pack("I", event.daddr))
    dport = socket.ntohs(event.dport)
    print(f"{event.pid:<8} {event.comm.decode():<16} {dip}:{dport}")

b["events"].open_perf_buffer(print_event)

while True:
    b.perf_buffer_poll()

#!/usr/bin/env python3
# slow_query_tracer.py — Find PostgreSQL queries taking > 100ms

from bcc import BPF
import time

bpf_program = """
#include <uapi/linux/ptrace.h>

#define MAX_QUERY_LEN 200

struct query_start_t {
    u64 start_ns;
    char query[MAX_QUERY_LEN];
};

// Key: thread ID, Value: when query started + query text
BPF_HASH(query_start, u32, struct query_start_t);

BPF_PERF_OUTPUT(slow_queries);

struct slow_query_t {
    u32 pid;
    u64 latency_ms;
    char query[MAX_QUERY_LEN];
};

// Probe on PostgreSQL's exec_simple_query function
int trace_query_start(struct pt_regs *ctx) {
    u32 tid = bpf_get_current_pid_tgid();
    struct query_start_t qs = {};
    qs.start_ns = bpf_ktime_get_ns();

    // Read query string from first argument
    bpf_probe_read_user_str(&qs.query, sizeof(qs.query), (void *)PT_REGS_PARM1(ctx));

    query_start.update(&tid, &qs);
    return 0;
}

int trace_query_end(struct pt_regs *ctx) {
    u32 tid = bpf_get_current_pid_tgid();
    struct query_start_t *qs = query_start.lookup(&tid);
    if (!qs) return 0;

    u64 latency_ms = (bpf_ktime_get_ns() - qs->start_ns) / 1000000;

    if (latency_ms > 100) {  // Only report queries > 100ms
        struct slow_query_t sq = {};
        sq.pid = bpf_get_current_pid_tgid() >> 32;
        sq.latency_ms = latency_ms;
        __builtin_memcpy(&sq.query, &qs->query, MAX_QUERY_LEN);
        slow_queries.perf_submit(ctx, &sq, sizeof(sq));
    }

    query_start.delete(&tid);
    return 0;
}
"""

b = BPF(text=bpf_program)

# Attach to PostgreSQL (uprobe on specific binary function)
PG_BINARY = "/usr/lib/postgresql/14/bin/postgres"
b.attach_uprobe(name=PG_BINARY, sym="exec_simple_query", fn_name="trace_query_start")
b.attach_uretprobe(name=PG_BINARY, sym="exec_simple_query", fn_name="trace_query_end")

print("Tracing PostgreSQL slow queries (>100ms)...")

def print_slow_query(cpu, data, size):
    event = b["slow_queries"].event(data)
    query = event.query.decode(errors='replace')
    print(f"[{event.latency_ms}ms] PID={event.pid}: {query[:100]}")

b["slow_queries"].open_perf_buffer(print_slow_query)
while True:
    b.perf_buffer_poll()

---

48.4 libbpf + CO-RE (Compile Once, Run Everywhere)

// minimal_bpf.c — Modern eBPF with libbpf and CO-RE
// Requires: kernel 5.8+, BTF enabled

#include "vmlinux.h"     // Auto-generated kernel types
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

// Ring buffer for kernel→user communication
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);  // 256KB buffer
} rb SEC(".maps");

struct event {
    __u32 pid;
    __u8 comm[16];
    __u64 latency_ns;
};

// Trace openat syscall
SEC("tp/syscalls/sys_enter_openat")
int handle_openat(struct trace_event_raw_sys_enter *ctx) {
    struct event *e;

    e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e)
        return 0;

    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    e->latency_ns = bpf_ktime_get_ns();

    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

---

48.5 Interview Questions

Q1: What makes eBPF safe to run in the kernel?

Answer: The eBPF verifier is the key safety mechanism. Before any eBPF program runs, the kernel verifier performs static analysis: (1) CFG analysis: ensures all code paths eventually terminate (no unbounded loops). (2) Bounds checking: every memory access is verified against safe ranges — no out-of-bounds reads/writes. (3) Pointer tracking: tracks pointer types to prevent type confusion. (4) Instruction limit: max 1 million instructions total. (5) Privileged helpers: eBPF programs can only call a curated set of kernel helpers, not arbitrary functions. After verification, the program is JIT-compiled to native machine code and runs with near-native performance. If verification fails, the program is rejected — it can never crash the kernel or access unauthorized memory.

Q2: What is the difference between kprobes and tracepoints for eBPF?

Answer: kprobes can attach to any kernel function dynamically — they work by inserting a breakpoint instruction at the target function entry. They’re flexible (any function) but fragile: function names can change between kernel versions, arguments can shift. Tracepoints are stable, explicitly-defined hooks in the kernel source code. They have a defined, versioned interface (documented struct of arguments). Tracepoints are preferred because: (1) Stable ABI — won’t break between kernel versions. (2) Lower overhead (no breakpoint mechanism). (3) Better supported for CO-RE (Compile Once, Run Everywhere). Use kprobes when there’s no tracepoint for what you need. Use tracepoints (prefixed tp/ in bpftrace) whenever available.

---

48.6 Summary

Tool	Use Case
bpftrace	One-liner tracing, ad-hoc debugging
BCC Python	Production tools with Python frontend
libbpf + CO-RE	Portable compiled eBPF tools
kprobes	Attach to any kernel function
uprobes	Attach to user-space functions
tracepoints	Stable kernel hook points
Ring buffer	High-performance kernel→user data

---

Next Chapter: Chapter 49: DPDK, XDP & Kernel Bypass Networking

Prerequisites

Chapter 27 (eBPF & XDP), C programming.

---

Advantages & Disadvantages

Advantages: The ultimate Linux superpower for observability and security, safe by design. Disadvantages: CO-RE/BTF is complex to set up, still evolving rapidly.

---

Common Mistakes

• Using BCC for production tools (it requires LLVM on the production machine). Use libbpf (CO-RE) instead.
• Writing unbounded loops — the verifier will reject the program.
• Assuming eBPF can arbitrarily modify kernel memory — it’s strictly read-only for most kernel structures.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Verifier rejects program (instruction limit exceeded)	Program too complex or loop unrolling failed	Check verifier log	Simplify logic, use tail calls to chain multiple eBPF programs together
BPF map updates failing	Map size exceeded	Check bpf_map_update_elem return value	Increase max_entries when creating the BPF map

---

Hands-on Lab

Objective: Use a bpftrace one-liner.

# 1. Count syscalls by program (run for 10 seconds, then Ctrl+C)
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

---

Exercises

1. Write a bpftrace script that measures the latency of the read() system call and prints a histogram of the latencies.
2. Research CO-RE (Compile Once, Run Everywhere) and explain how BTF (BPF Type Format) solves the kernel version dependency problem.

---

Revision Notes

• eBPF uses Maps (Hash, Array, Ring Buffer) to share state between the kernel program and user space.
• CO-RE and BTF allow compiling an eBPF program once and running it on different kernel versions.
• BCC is great for prototyping; libbpf is the standard for production deployment.
• Tail calls allow bypassing the 1-million instruction limit by jumping to another eBPF program.

---

Further Reading

• BPF Performance Tools by Brendan Gregg
• Cilium eBPF documentation

---

Related Chapters

• Chapter 27 — eBPF Basics

---

Last Updated: July 2026