strace

Chapter 87: strace, ltrace, and syscalls - Deep Dive

Mastering System Call Tracing for Troubleshooting

Why This Matters in DevOps/SRE

strace is essential for debugging application issues:

Deep Troubleshooting: See exactly what system calls an application makes
Performance: Identify slow syscalls causing delays
Debugging: Find missing files, permission issues, network problems
Security: Detect suspicious system call patterns
Root Cause: Get to the actual failure, not just symptoms

strace is the go-to tool when logs don’t provide enough information.

87.1 Understanding System Calls

What are System Calls?

System calls (syscalls) are the fundamental interface between user applications and the Linux kernel. They provide a way for programs to request services from the kernel.

                     System Call Flow
+------------------------------------------------------------------+
|                                                                   |
|    User Space                                Kernel Space          |
|                                                                   |
|    +----------------+                                           |
|    | Application    |                                           |
|    +--------+-------+                                           |
|             |                                                    |
|             | syscall instruction                                |
|             v                                                    |
|    +--------+-------+                                           |
|    |   libc (glibc) |  - syscall wrapper                      |
|    +--------+-------+                                           |
|             |                                                    |
|             | system call                                        |
|             v                                                    |
|    +-------------------------------------------------+         |
|    |              System Call Interface                   |         |
|    |  - Validates parameters                             |         |
|    |  - Switches to kernel mode                         |         |
|    |  - Routes to appropriate handler                    |         |
|    +-------------------------------------------------+         |
|             |                                                    |
|             v                                                    |
|    +-------------------------------------------------+         |
|    |              Kernel Subsystem                        |         |
|    |  - File system                                    |         |
|    |  - Process management                             |         |
|    |  - Memory management                             |         |
|    |  - Network                                      |         |
|    +-------------------------------------------------+         |
|                                                                   |
+------------------------------------------------------------------+

Common System Calls

                     Common System Calls
+------------------------------------------------------------------+
|                                                                   |
|    Process Management:                                            |
|    +----------------------------------------------------------+   |
|    |  fork()     - Create new process                       |   |
|    |  execve()   - Execute program                          |   |
|    |  waitpid()  - Wait for process                        |   |
|    |  exit()     - Terminate process                       |   |
|    |  getpid()   - Get process ID                          |   |
|    |  getppid()  - Get parent process ID                   |   |
|    |  nice()     - Change process priority                 |   |
|    |  kill()     - Send signal to process                  |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    File Operations:                                               |
|    +----------------------------------------------------------+   |
|    |  open()     - Open file                               |   |
|    |  close()    - Close file                              |   |
|    |  read()     - Read from file                         |   |
|    |  write()    - Write to file                          |   |
|    |  lseek()    - Change file position                   |   |
|    |  stat()     - Get file status                        |   |
|    |  chmod()    - Change file permissions                 |   |
|    |  rename()   - Rename file                            |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Memory:                                                       |
|    +----------------------------------------------------------+   |
|    |  mmap()     - Map memory                              |   |
|    |  munmap()   - Unmap memory                           |   |
|    |  brk()      - Change data segment size               |   |
|    |  mprotect() - Set memory protection                  |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Network:                                                       |
|    +----------------------------------------------------------+   |
|    |  socket()   - Create socket                          |   |
|    |  bind()     - Bind socket to address                |   |
|    |  listen()   - Listen for connections                |   |
|    |  accept()   - Accept connection                    |   |
|    |  connect()  - Connect to socket                    |   |
|    |  send()     - Send data                            |   |
|    |  recv()     - Receive data                         |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Information:                                                   |
|    +----------------------------------------------------------+   |
|    |  getuid()   - Get user ID                          |   |
|    |  getgid()   - Get group ID                         |   |
|    |  uname()    - Get system information               |   |
|    |  time()     - Get current time                     |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

87.2 strace Fundamentals

Using strace

# =============================================================================
# BASIC USAGE
# =============================================================================

# Trace new process
strace ls -la

# Attach to running process
strace -p 1234

# Trace with timing
strace -t ls -la
strace -tt ls -la        # Microseconds
strace -T ls -la        # Time in syscall

# =============================================================================
# OUTPUT OPTIONS
# =============================================================================

# Redirect to file
strace -o output.txt ls

# Append to file
strace -a output.txt ls

# Show instruction pointer
strace -i ls

# Show relative time
strace -r ls

# Compact output
strace -q ls

# Color output
strace -C ls

# =============================================================================
# FILTERING
# =============================================================================

# Trace specific syscalls
strace -e trace=open,read,write ls

# Trace by category
strace -e trace=file ls
strace -e trace=network ls
strace -e trace=process ls
strace -e trace=memory ls
strace -e trace=signal ls
strace -e trace=ipc ls

# Trace all except
strace -e trace=!read ls

# Trace file creation
strace -e trace=creat ls

# =============================================================================
# FOLLOW FORK
# =============================================================================

# Follow child processes
strace -f ls

# Trace fork+exec
strace -f -e trace=execve bash -c "ls"

# =============================================================================
# CONDITIONAL TRACING
# =============================================================================

# Trace after condition
strace -P /path/to/file ls

# Filter by result
strace -e trace=open -e success ls

# Filter by error
strace -e trace=open -e failed ls

# =============================================================================
# STRACE STATISTICS
# =============================================================================

# Summary of syscalls
strace -c ls

# Summary with errors
strace -c -e trace=open,read,write ls

# Print time summary
strace -c -w ls

# =============================================================================
# DIAGNOSTIC OPTIONS
# =============================================================================

# Print raw syscalls
strace -n ls

# Print command name
strace -s ls

# Help diagnose crashes
strace -o crash.txt -f -g command

# Show failed calls only
strace -z ls

# =============================================================================
# PROCESS OPTIONS
# =============================================================================

# Read from stdin
strace -I 2 ls

# Trace children on fork
strace -f ls

# Trace threads
strace -f -ff ls

87.3 Practical strace Examples

Troubleshooting Examples

# =============================================================================
# DEBUGGING FILE ACCESS
# =============================================================================

# Find what files a program opens
strace -e trace=open,openat,close ls 2>&1 | grep -v "ENOENT"

# Trace specific file access
strace -e trace=open -P /etc/passwd cat /etc/passwd

# Watch for config file changes
strace -f -e trace=open,write nginx

# =============================================================================
# DEBUGGING NETWORK
# =============================================================================

# Trace network calls
strace -e trace=connect,accept,send,recv,close -f nginx

# Trace specific port
strace -e trace=network -p $(pgrep -f "port:8080")

# =============================================================================
# DEBUGGING PERFORMANCE
# =============================================================================

# Find slow syscalls
strace -T -c ls

# Time per syscall
strace -T ls

# Slow calls only (>1ms)
strace -T -s 100 ls 2>&1 | grep -v "0.000" | head -20

# =============================================================================
# DEBUGGING PERMISSIONS
# =============================================================================

# Trace permission errors
strace -e trace=access,open ls /root

# Trace failed opens
strace -e trace=open -z ls /nonexistent

# =============================================================================
# DEBUGGING PROCESS
# =============================================================================

# Find what process is doing
strace -p 1234 -c

# Trace system calls
strace -p 1234 -f

# =============================================================================
# DEBUGGING STARTUP
# =============================================================================

# Full trace with timing
strace -t -f -T -o /tmp/strace.log nginx

# Summary at exit
strace -c -f nginx

87.4 ltrace - Library Tracing

Using ltrace

# =============================================================================
# BASIC USAGE
# =============================================================================

# Trace library calls
ltrace ls

# Trace with timing
ltrace -t ls

# Trace with relative time
ltrace -r ls

# =============================================================================
# OUTPUT OPTIONS
# =============================================================================

# Output to file
ltrace -o output.txt ls

# Color output
ltrace -C ls

# =============================================================================
# FILTERING
# =============================================================================

# Trace specific library
ltrace -e memcpy ls

# Trace by regex
ltrace -e '*alloc*' ls

# Trace by library
ltrace -l libc.so.6 ls

# Exclude
ltrace -e '!*malloc*' ls

# =============================================================================
# STATISTICS
# =============================================================================

# Summary
ltrace -c ls

# Time summary
ltrace -c -A 10 ls

# =============================================================================
# ATTACH TO PROCESS
# =============================================================================

# Attach to running process
ltrace -p 1234

# =============================================================================
# COMMON OPTIONS
# =============================================================================

# Follow fork
ltrace -f ls

# Align columns
ltrace -A 20 ls

# Library path
ltrace -L /lib ls

87.5 syscall Reference

Common System Calls by Category

# =============================================================================
# FILE OPERATIONS
# =============================================================================

# open, openat - Open file
# close - Close file
# read, pread - Read from file
# write, pwrite - Write to file
# lseek - Reposition read/write file offset
# stat, fstat, lstat - Get file status
# access, faccessat - Check file access
# chmod, fchmod - Change file permissions
# chown, fchown - Change file owner
# rename - Rename file
# mkdir - Create directory
# rmdir - Remove directory
# unlink, unlinkat - Remove file
# link, linkat - Create hard link
# symlink, readlink - Symbolic links

# =============================================================================
# PROCESS MANAGEMENT
# =============================================================================

# fork - Create process
# vfork - Create process (optimized)
# clone - Create thread/process
# execve - Execute program
# _exit - Terminate process
# wait, waitpid, wait4 - Wait for process
# getpid, getppid - Get process ID
# getuid, geteuid - Get user ID
# getgid, getegid - Get group ID
# setuid, setgid - Set user/group ID
# nice - Change priority
# sched_setscheduler, sched_getscheduler - CPU scheduling
# kill - Send signal
# alarm - Set alarm clock

# =============================================================================
# MEMORY
# =============================================================================

# brk - Change data segment size
# mmap, munmap - Map/unmap memory
# mprotect - Set memory protection
# mlock, munlock - Lock memory
# mremap - Remap memory
# madvise - Give advice about memory

# =============================================================================
# NETWORKS
# =============================================================================

# socket - Create socket
# bind - Bind socket to address
# listen - Listen for connections
# accept, accept4 - Accept connection
# connect - Connect to socket
# send, sendto, sendmsg - Send data
# recv, recvfrom, recvmsg - Receive data
# shutdown - Shutdown socket
# getsockopt, setsockopt - Socket options

# =============================================================================
# IPC
# =============================================================================

# msgget - Get message queue
# msgsnd, msgrcv - Message operations
# semget - Get semaphore set
# semop, semtimedop - Semaphore operations
# shmget - Get shared memory
# shmat, shmdt - Attach/detach shared memory

# =============================================================================
# TIME
# =============================================================================

# time - Get current time
# gettimeofday - Get time with microseconds
# clock_gettime - Get clock time
# nanosleep - High-resolution sleep

# =============================================================================
# SYSTEM
# =============================================================================

# uname - Get system information
# sysinfo - Get system info
# getrlimit, setrlimit - Resource limits
# getrusage - Get resource usage
# syslog - Read/write kernel messages
# gettid - Get thread ID
# set_thread_area - Thread local storage

87.6 Performance Analysis

Analyzing Performance Issues

# =============================================================================
# FIND SLOW OPERATIONS
# =============================================================================

# Time spent in syscalls
strace -c -w ls

# Find slow syscalls (>1ms)
strace -T -s 100 command 2>&1 | awk -F'=' '/0\.[0-9]+/ && $2 > 0.001 {print}'

# Analyze with perf
perf record -g ./command
perf report

# =============================================================================
# STRACE STATISTICS ANALYSIS
# =============================================================================

# Full statistics
strace -c -f ./script.sh

# Percentage-based
strace -c -f --sort=time ./script.sh

# =============================================================================
# REAL-TIME ANALYSIS
# =============================================================================

# Monitor live
strace -p 1234 -c -e trace=write

# Trace specific duration
timeout 10 strace -p 1234 -c

87.7 Exam Tips

strace: Trace system calls (kernel)
ltrace: Trace library calls (userspace)
Filters: Use -e trace= for specific syscalls
Timing: -T shows time per syscall
Statistics: -c summarizes all calls
Attach: -p for running processes
Fork: -f follows child processes
Output: -o saves to file
Errors: -z shows failed calls only
Performance: Use -T to find slow operations

Common Mistakes & Anti-Patterns

1. Tracing Without Filters

WRONG:

# Tracing everything - overwhelming output
strace -p $(pgrep nginx)

CORRECT:

# Filter specific syscalls
strace -p $(pgrep nginx) -e trace=open,read,write
strace -p $(pgrep nginx) -e trace=network
strace -p $(pgrep nginx) -e trace=file

Why: Unfiltered output is unreadable and consumes massive disk space.

2. Not Using Timestamps

WRONG:

# No timing information
strace ./script.sh

CORRECT:

# Add timing
strace -t ./script.sh          # Relative time
strace -tt ./script.sh         # Microseconds
strace -r ./script.sh          # Relative time per call
strace -T ./script.sh          # Time spent in call

Why: Timestamps identify slow operations.

3. Running in Production Without Care

WRONG:

# Tracing production service with full output
strace -p $(pgrep nginx) > /var/log/strace.log
# System slows to a crawl!

CORRECT:

# Use minimal overhead options
strace -p $(pgrep nginx) -e trace=write -q
strace -c -p $(pgrep nginx)  # Summary only

# Or use BPF-based tools instead
perf record -g -p $(pgrep nginx)

Why: strace adds significant overhead.

4. Not Following Child Processes

WRONG:

# Only tracing main process
strace -p $(pgrep nginx)
# Missing fork() children

CORRECT:

# Follow forks
strace -f -p $(pgrep nginx)
strace -ff -p $(pgrep nginx)  # Separate files per child

Why: Many services fork child processes.

5. Ignoring Return Values

WRONG:

# Not checking for errors
strace -e trace=open ./script
# Just looking at what opens

CORRECT:

# Show failed calls
strace -e trace=open,openat ./script 2>&1 | grep -v "= -1"

# Show only errors
strace -z -p $(pgrep nginx)  # Show only successful calls
strace -Z -p $(pgrep nginx)  # Show only failed calls

Why: Failed syscalls often indicate the actual problem.

Summary

In this chapter, you learned:

✅ System call concepts and architecture
✅ strace fundamentals and usage
✅ Common system calls
✅ Practical troubleshooting examples
✅ ltrace for library calls
✅ Performance analysis with strace
✅ Best practices

Next Chapter

Chapter 88: tcpdump and Wireshark

Last Updated: February 2026