Signals, IPC, epoll & File Descriptors

Chapter 6: Signals, IPC, epoll & File Descriptors

Learning Objectives

Understand file descriptors and how everything in Linux is a file
Understand IPC mechanisms and when to use each
Understand epoll and how high-performance servers handle thousands of connections
Know how to diagnose file descriptor leaks in production

Prerequisites

What is this? (Beginner On-Ramp)

Signals and IPC (Inter-Process Communication) are how running programs talk to each other. A signal is like tapping someone on the shoulder to get their attention (e.g., ‘stop running!’). IPC involves setting up pipes or shared memory so programs can actually pass data back and forth.

6.1 Everything is a File: The Unix Philosophy

In Linux, everything is a file — or more precisely, everything is accessed through a file descriptor (fd).

    Types of File Descriptors
    ──────────────────────────
    Regular Files: /etc/nginx/nginx.conf
    Directories: /var/log/
    Pipes: output | input
    Sockets: TCP/UDP/Unix domain sockets
    Terminals: /dev/tty
    Devices: /dev/sda, /dev/null, /dev/random
    Proc entries: /proc/1234/maps
    Eventfd: signaling between processes
    Timerfd: timer notifications
    Signalfd: receive signals as file reads
    Epollfd: I/O event monitoring

    All accessed via the same API:
    open(), read(), write(), close(), ioctl()

The File Descriptor Table

    Process File Descriptor Table
    ──────────────────────────────
    PID=1234 (nginx worker)
    ┌─────┬───────────────────────────────────────────┐
    │  fd │  Points To                                │
    ├─────┼───────────────────────────────────────────┤
    │   0 │  stdin  → /dev/null (daemons discard it)  │
    │   1 │  stdout → /dev/null (or log file)         │
    │   2 │  stderr → /var/log/nginx/error.log        │
    │   3 │  listening socket (TCP port 80)            │
    │   4 │  listening socket (TCP port 443)           │
    │   5 │  epoll file descriptor                     │
    │   6 │  /var/cache/nginx/... (temp file)          │
    │   7 │  connection socket (client request)        │
    │   8 │  connection socket (another client)        │
    │  .. │  ...                                       │
    │1023 │  last allowed fd (default limit)           │
    └─────┴───────────────────────────────────────────┘

# View open file descriptors for a process
ls -la /proc/<PID>/fd/

# Count open FDs
ls /proc/<PID>/fd/ | wc -l

# See FD limits
cat /proc/<PID>/limits | grep "open files"
# Max open files  65536  65536  files

# Increase global FD limit
echo "fs.file-max = 2097152" >> /etc/sysctl.conf
sysctl -p

# Increase per-process limit (in /etc/security/limits.conf):
* soft nofile 65536
* hard nofile 131072

# In systemd service:
LimitNOFILE=131072

# Find which process has the most FDs open
lsof 2>/dev/null | awk '{print $2}' | sort | uniq -c | sort -rn | head -10

File Descriptor Leaks

# Monitor for FD leaks (process should not have monotonically growing FD count)
while true; do
    FD_COUNT=$(ls /proc/<PID>/fd 2>/dev/null | wc -l)
    echo "$(date): FD count = $FD_COUNT"
    sleep 30
done

# Identify leaked FDs
lsof -p <PID> | awk '{print $4, $5, $9}' | sort

# Common causes:
# 1. Opening files without closing them
# 2. Forking without closing parent FDs (FD_CLOEXEC)
# 3. Accepting sockets without closing them on error

# Good practice: set FD_CLOEXEC on all FDs
# In C: fcntl(fd, F_SETFD, FD_CLOEXEC)
# In Python: socket.SOCK_CLOEXEC flag
# Prevents FD leaks through fork+exec

6.2 IPC: Inter-Process Communication

Linux provides multiple mechanisms for processes to communicate. Choose the right one based on performance, simplicity, and use case.

    IPC Mechanism Comparison
    ──────────────────────────

    ┌──────────────────┬────────────┬───────────┬───────────────────────┐
    │   Mechanism      │  Speed     │ Use Case  │  Notes                │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ Pipe             │ Medium     │ Parent→   │ Unidirectional, same  │
    │                  │            │ Child      │ host only             │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ Named Pipe (FIFO)│ Medium     │ Any procs │ Bidirectional by      │
    │                  │            │ same host  │ using two FIFOs      │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ Unix Socket      │ Fast       │ Any procs │ Full duplex, same host│
    │                  │            │ same host  │ MySQL, nginx→app     │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ TCP Socket       │ Medium     │ Any procs │ Cross-host, overhead  │
    │                  │            │ any host   │ of TCP stack         │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ Shared Memory    │ Fastest    │ High perf │ Requires sync (mutex) │
    │ (mmap/shm)       │            │ same host  │ Redis, huge pages    │
    ├──────────────────┼────────────┼───────────┼───────────────────────┤
    │ Message Queue    │ Medium     │ Async msg │ Ordered delivery      │
    │ (POSIX mq)       │            │            │ kernel-buffered      │
    └──────────────────┴────────────┴───────────┴───────────────────────┘

Pipes

# Classic pipe usage
ls -la /etc | grep nginx

# The kernel connects stdout of ls to stdin of grep
# Via an anonymous pipe (kernel buffer, typically 64KB)

# Named pipe (FIFO)
mkfifo /tmp/mypipe

# Terminal 1: Write to pipe (blocks until someone reads)
echo "hello from process A" > /tmp/mypipe

# Terminal 2: Read from pipe
cat /tmp/mypipe

# Clean up
rm /tmp/mypipe

Unix Domain Sockets

Unix sockets are faster than TCP because there’s no TCP/IP stack overhead:

# Many applications use Unix sockets internally:
# MySQL: /var/run/mysqld/mysqld.sock
# Docker: /var/run/docker.sock

# Check which sockets exist
ls -la /var/run/*.sock

# Connect to a Unix socket for debugging
nc -U /var/run/docker.sock

# Monitor Unix socket traffic
strace -e trace=recvfrom,sendto -p <PID>

Shared Memory

# View existing shared memory segments
ipcs -m

# Create shared memory in a program:
# POSIX: shm_open(), mmap()
# SysV: shmget(), shmat()

# Redis uses shared memory for high performance
# PostgreSQL uses shared_buffers in shared memory
ipcs -m | grep postgres

# In production: shared memory segments can accumulate
# after process crashes without cleanup
# Fix: ipcrm -M <shmid>

6.3 epoll: The Foundation of High-Performance I/O

epoll is how every high-performance server (nginx, Node.js, Redis, etc.) handles thousands of concurrent connections with minimal CPU.

The C10K Problem

    The Problem: 10,000 Simultaneous Connections
    ─────────────────────────────────────────────

    Naive approach: One thread per connection
    ─────────────────────────────────────────
    10,000 connections = 10,000 threads
    Each thread: 8MB stack = 80GB RAM just for stacks!
    Context switching overhead: catastrophic

    Old select()/poll() approach:
    ─────────────────────────────
    for each connection check:
        if ready? → handle it
        else → skip
    O(n) scan of all FDs on every check
    10,000 FDs = 10,000 checks every call

    epoll approach:
    ──────────────
    Kernel maintains a ready list
    Application asks: "what's ready now?"
    Only ready FDs are returned
    O(1) regardless of number of watched FDs
    10,000 FDs with 10 active = return 10, not 10,000

How epoll Works

    epoll Operation
    ───────────────

    1. Create epoll instance:
       epfd = epoll_create1(0)
       → Returns a file descriptor for the epoll set

    2. Add sockets to watch:
       epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &event)
       → Registers sockfd with the kernel's wait queue

    3. Wait for events:
       n = epoll_wait(epfd, events, MAX_EVENTS, timeout_ms)
       → Blocks until at least one FD is ready
       → Returns events[] containing ready FDs

    4. Handle only ready FDs:
       for i in range(n):
           if events[i].data.fd == listen_fd:
               new_conn = accept()
               epoll_ctl(ADD, new_conn, &event)
           else:
               read/write to events[i].data.fd

    ┌─────────────────────────────────────────────────────┐
    │               epoll Architecture                     │
    │                                                      │
    │  Application                                         │
    │  ┌──────────┐  epoll_wait()  ┌───────────────────┐  │
    │  │ Event    │◄───────────────│  Kernel epoll     │  │
    │  │ Loop     │                │  Event Queue      │  │
    │  └──────────┘                │                   │  │
    │        │                     │  [fd=7, EPOLLIN]  │  │
    │        │ handle              │  [fd=12, EPOLLOUT]│  │
    │        ▼                     │                   │  │
    │  ┌────────────┐              └────────┬──────────┘  │
    │  │read(fd=7)  │                       │ kernel       │
    │  │send(fd=12) │              ┌────────┘ notifies     │
    │  └────────────┘              │          when ready   │
    │                    ┌─────────▼─────────────────────┐ │
    │                    │     Watched FDs               │ │
    │                    │  fd=3 (listen socket)         │ │
    │                    │  fd=7 (client conn 1)         │ │
    │                    │  fd=12 (client conn 2)        │ │
    │                    │  fd=15 (client conn 3)        │ │
    │                    │  ... up to millions           │ │
    │                    └───────────────────────────────┘ │
    └──────────────────────────────────────────────────────┘

Edge vs Level Triggered

    EPOLLET (Edge Triggered) vs default (Level Triggered)
    ──────────────────────────────────────────────────────

    Level Triggered (default):
    epoll_wait returns ready FDs as long as data is available
    → Simpler to use (can't miss data)
    → Each epoll_wait call may return the same FD multiple times

    Edge Triggered (EPOLLET):
    epoll_wait returns ready FDs only when state CHANGES
    → FD becomes readable: returned ONCE
    → Must read ALL available data before epoll_wait again
    → Requires non-blocking I/O (EAGAIN means "read it all")
    → Higher performance (fewer epoll_wait wakeups)
    → Used by nginx, Redis

    nginx epoll configuration:
    events {
        use epoll;
        worker_connections 10240;
        multi_accept on;      # Accept multiple connections per epoll event
    }

6.4 File Descriptor Limits in Production

# The single most common "nginx won't start" error:
# "too many open files" or "accept4() failed (24: Too many open files)"

# Check current system-wide FD usage
cat /proc/sys/fs/file-nr
# 12345 0 2097152
# │     │ └── max FDs system-wide
# │     └──── FDs marked for closing (legacy, always 0)
# └────────── currently open FDs

# Production FD limit configuration:
# 1. System-wide
echo "fs.file-max = 2097152" > /etc/sysctl.d/99-fd.conf
sysctl -p /etc/sysctl.d/99-fd.conf

# 2. Per-user (applies to login sessions)
cat /etc/security/limits.conf
# * soft nofile 65536
# * hard nofile 131072
# www-data soft nofile 65536

# 3. systemd services (overrides limits.conf for services)
# In /etc/systemd/system/nginx.service:
[Service]
LimitNOFILE=65536

# Verification
sudo -u www-data bash -c 'ulimit -n'
systemctl show nginx | grep LimitNOFILE
cat /proc/$(pidof nginx | awk '{print $1}')/limits | grep "Max open files"

6.5 Production Scenarios

Scenario 1: “Too Many Open Files” Error

# Problem: nginx workers failing with "too many open files"

# Step 1: Check current limits
cat /proc/$(pgrep -f "nginx: worker" | head -1)/limits | grep "open files"

# Step 2: Check how many FDs are actually open
ls /proc/$(pgrep -f "nginx: worker" | head -1)/fd | wc -l

# Step 3: What are all these FDs?
lsof -p $(pgrep -f "nginx: worker" | head -1) | awk '{print $5}' | sort | uniq -c

# Step 4: Fix
# In /etc/nginx/nginx.conf:
# worker_rlimit_nofile 65536;

# In /etc/systemd/system/nginx.service:
# LimitNOFILE=65536

sudo systemctl daemon-reload
sudo systemctl restart nginx

Scenario 2: Node.js Application with High FD Count

# Node.js app has 10,000 open connections and FD count approaches limit

# Step 1: Current state
ls /proc/$(pidof node)/fd | wc -l
cat /proc/$(pidof node)/limits | grep "Max open files"

# Step 2: What are all the FDs?
lsof -p $(pidof node) | grep -c TCP   # Count TCP sockets
lsof -p $(pidof node) | grep -c IPv4  # Count IPv4 connections

# Step 3: Are there connection leaks?
# If keep-alive connections are not being closed properly,
# the FD count will grow monotonically

# Step 4: Fix the app
# Set keepAlive timeout properly
# In Express: server.keepAliveTimeout = 65000;
# Ensure client sockets are destroyed on error

# Step 5: Increase limit as band-aid
sudo systemctl set-property mynode.service LimitNOFILE=1000000

6.6 Interview Questions

Q1: What is the difference between a process and a file descriptor from the kernel’s perspective?

Answer: A process is represented by a task_struct containing execution context, memory maps, and a file descriptor table. A file descriptor is just an integer index into the per-process FD table, which contains pointers to struct file objects in the kernel. A struct file tracks the offset, access mode, and a pointer to the underlying inode or socket. The same struct file can be referenced by multiple FDs (via dup()) or by multiple processes (inherited via fork()). When all references are closed, the struct file is freed.

Q2: How does epoll achieve O(1) scalability while poll() is O(n)?

Answer: poll() requires the application to pass all watched FDs on every call, and the kernel must scan all of them to check their readiness — O(n). epoll maintains a kernel-side data structure (a red-black tree) of watched FDs. When a FD becomes ready, the kernel moves it to a ready list via a callback mechanism. epoll_wait() just returns items from the ready list — O(1) regardless of the total number of watched FDs. This is why nginx can handle 10,000 connections with low CPU: it’s notified only when a connection has data, not polling all connections continuously.

Q3: What happens when you run out of file descriptors in production?

Answer: Once the FD count reaches the process limit (default 1024 on older systems, configurable), all subsequent calls to open(), socket(), accept(), dup(), etc. return ENOFILE (24: Too many open files). For a web server like nginx, this means it can’t accept new connections. For an application, it can’t open new files or sockets. The system doesn’t crash — existing FDs still work. Fix by: (1) increasing limits via LimitNOFILE in the systemd service, (2) fixing FD leaks in the application, (3) implementing connection pooling to reuse FDs.

6.7 Summary

Concept	Key Point
File Descriptor	Integer index into process’s open file table
Everything is a file	Sockets, pipes, devices all use same fd API
Pipe	Unidirectional kernel buffer between processes
Unix Socket	Faster than TCP for same-host communication
Shared Memory	Fastest IPC, requires explicit synchronization
epoll	O(1) event notification, powers all high-perf servers
FD Limits	Default 1024; increase to 65536+ for servers

Next Chapter: Chapter 7: systemd Deep Dive

Prerequisites

Chapter 3 (Process Lifecycle) — understand processes and process states.

Advantages & Disadvantages

Advantages: Standardized communication, shared memory provides zero-copy high performance. Disadvantages: Signals interrupt program flow unexpectedly, IPC mechanisms (like semaphores) are prone to race conditions.

Common Mistakes

Assuming all signals can be caught — SIGKILL (9) and SIGSTOP (19) are uncatchable by design; they’re kernel-enforced.
Calling non-async-signal-safe functions inside signal handlers — printf() is NOT safe in signal handlers; use write() instead.
Forgetting epoll’s edge-triggered mode requires draining the entire buffer — in ET mode, you MUST read until EAGAIN or you miss events.
Using select() for high-fd counts — select() has a hard limit of FD_SETSIZE (1024). Use epoll for >1000 connections.
Closing a pipe’s read end before writing finishes — the writer gets SIGPIPE; unhandled it crashes the process.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Process ignoring SIGTERM	Signal handler set to SIG_IGN, or handler doesn’t exit	`strace -e signal -p PID to see signal delivery`	Fix app to handle SIGTERM; as last resort: SIGKILL
epoll returning EBADF	File descriptor closed while still in epoll interest list	`strace -e epoll_wait,close -p PID`	Always call epoll_ctl(EPOLL_CTL_DEL) before close(); or use EPOLLONESHOT
Broken pipe errors in network app	Writer sending to closed socket	`dmesg; application logs for SIGPIPE`	Signal(SIGPIPE, SIG_IGN) in app; check return value of send()/write()
Message queue full — producer blocked	Consumer too slow or dead	`ipcs -q; check msg_qnum vs msg_qbytes`	Increase queue size via /proc/sys/kernel/msgmnb; fix consumer; use async queue

Hands-on Lab

Objective: Explore signals, pipes, and epoll.

# 1. List all signals
kill -l

# 2. Trap and handle SIGTERM in bash
trap 'echo "Got SIGTERM, cleaning up..."; exit 0' SIGTERM
sleep 100 &
PID=$!
kill -SIGTERM $PID
wait $PID

# 3. Pipe communication
(echo "hello from writer") | (read msg; echo "Reader got: $msg")

# 4. Named pipe (FIFO)
mkfifo /tmp/myfifo
echo "data" > /tmp/myfifo &
cat /tmp/myfifo

# 5. Inspect open file descriptors for a process
ls -la /proc/$$/fd/

# 6. Check IPC objects (queues, semaphores, shared mem)
ipcs -a

# 7. Count epoll file descriptors (for a web server)
ls /proc/$(pgrep nginx | head -1)/fd | wc -l 2>/dev/null || echo "nginx not running"

Exercises

Write a C program that sets up a signal handler for SIGINT that prints ‘Caught SIGINT’ and increments a counter. After 3 catches, exit cleanly.
Create a producer-consumer model using a named pipe: one script writes numbers 1-100, another reads and prints them.
Build a minimal epoll server in C or Python that accepts TCP connections on port 8080 and echoes data back. Test with nc.
Implement a simple shared memory IPC: one process writes a message to shared memory, another reads it.
Explain why printf() is not safe in signal handlers. Write a signal-safe equivalent using write(2) directly.

Revision Notes

Signals are async notifications to processes: kill(PID, SIG) sends a signal; handler or default action runs.
SIGKILL and SIGSTOP cannot be caught, blocked, or ignored — they’re kernel-enforced.
File descriptors are integers indexing the kernel’s open file table — 0=stdin, 1=stdout, 2=stderr.
Pipes are unidirectional; sockets bidirectional. Both are FDs — read()/write() work on all.
epoll is O(1) for event notification regardless of number of FDs — unlike select() which is O(n).
Shared memory (shmget) is fastest IPC — zero copy. Message queues add ordering. Pipes are simplest.

Signals, IPC, epoll & File Descriptors

Chapter 6: Signals, IPC, epoll & File Descriptors

Learning Objectives

Prerequisites

What is this? (Beginner On-Ramp)

6.1 Everything is a File: The Unix Philosophy

The File Descriptor Table

File Descriptor Leaks

6.2 IPC: Inter-Process Communication

Pipes

Unix Domain Sockets

Shared Memory

6.3 epoll: The Foundation of High-Performance I/O

The C10K Problem

How epoll Works

Edge vs Level Triggered

6.4 File Descriptor Limits in Production

6.5 Production Scenarios

Scenario 1: “Too Many Open Files” Error

Scenario 2: Node.js Application with High FD Count

6.6 Interview Questions

6.7 Summary

Next Chapter: Chapter 7: systemd Deep Dive

Prerequisites

Advantages & Disadvantages

Common Mistakes

Troubleshooting

Hands-on Lab

Exercises

Revision Notes

Further Reading

Related Chapters