Core_dumps
Chapter 89: Core Dump Analysis
Section titled “Chapter 89: Core Dump Analysis”Overview
Section titled “Overview”Core dumps are memory snapshots of crashed processes that are invaluable for debugging production issues. When a process crashes unexpectedly, a core dump captures the entire state of the process at the time of the crash, including stack traces, variable values, and memory contents. This chapter covers enabling core dumps, analyzing them with gdb and other tools, kernel core dumps, and production best practices.
89.1 Enabling Core Dumps
Section titled “89.1 Enabling Core Dumps”Core Dump Fundamentals
Section titled “Core Dump Fundamentals”┌─────────────────────────────────────────────────────────────────────────┐│ CORE DUMP BASICS │├─────────────────────────────────────────────────────────────────────────┤│ ││ What is a Core Dump? ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ │ ││ │ A core dump is a file containing: │ ││ │ - Complete memory image of the process │ ││ │ - CPU register values at time of crash │ ││ │ - Stack traces for all threads │ ││ │ - Information about loaded libraries │ ││ │ - Process metadata (PID, user, etc.) │ ││ │ │ ││ │ Generated when process: │ ││ │ - Crashes with signal (SIGSEGV, SIGABRT, etc.) │ ││ │ - Receives signal with core action │ ││ │ - Calls abort() │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Core Dump Types: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ │ ││ │ 1. Core Dump (traditional) │ ││ │ - Single file, process memory only │ ││ │ │ ││ │ 2. Kernel Core Dump (vmcore) │ ││ │ - Entire kernel memory + processes │ ││ │ - Used for kernel crashes │ ││ │ │ ││ │ 3. Core Dump with Abbrev (gcore) │ ││ │ - Can attach to running process │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘System Configuration
Section titled “System Configuration”# ============================================================# ENABLING CORE DUMPS - SYSTEM CONFIGURATION# ============================================================
# /etc/security/limits.conf# Set core dump size limits for all users* soft core unlimited* hard core unlimitedroot soft core unlimitedroot hard core unlimited
# /etc/sysctl.conf - Kernel settings# Core pattern (where to save, what name)kernel.core_pattern = core.%p # %p = process IDkernel.core_uses_pid = 1 # Include PID in name
# Alternative patterns:# core.%e.%p - executable name + PID# core.%t - timestamp# core.%h - hostname
# Enable compressed core dumpskernel.core_compress_press = 1
# Set core dump size limit (0 = disabled)kernel.core_limit = 0 # 0 = unlimited in modern kernels
# Apply without rebootsudo sysctl -p
# Check current settingssysctl kernel.core_patternsysctl kernel.core_uses_pid
# System-wide ulimit (systemd)# /etc/systemd/system.confDefaultLimitCORE=infinityProcess Level Configuration
Section titled “Process Level Configuration”# ============================================================# PROCESS-LEVEL CORE DUMP CONFIGURATION# ============================================================
# In shell - enable for current sessionulimit -c unlimited
# In shell script#!/bin/bashulimit -c unlimited./my_program
# In Pythonimport resourcesoft, hard = resource.getrlimit(resource.RLIMIT_CORE)resource.setrlimit(resource.RLIMIT_CORE, (hard, hard))
# In C/C++ program#include <sys/resource.h>
struct rlimit rlim;rlim.rlim_cur = RLIM_INFINITY;rlim.rlim_max = RLIM_INFINITY;setrlimit(RLIMIT_CORE, &rlim);
# Using prctl (Linux-specific)#include <sys/prctl.h>prctl(PR_SET_COREFILE, PR_SET_COREFILE_LARGE);
# Disable core dumps for sensitive processesprctl(PR_SET_DUMPABLE, 0);
# Check if core dumps are enabled for a processcat /proc/<PID>/limits | grep "Max core file size"Application Configuration
Section titled “Application Configuration”# ============================================================# APPLICATION CORE DUMP CONFIGURATION# ============================================================
# For system services (systemd)# /etc/systemd/system/myapp.service[Service]LimitCORE=infinityWorkingDirectory=/path/to/appEnvironment="MALLOC_CHECK_=3" # Debug malloc issues
# For nginx# /etc/nginx/nginx.confworker_rlimit_core 100M;working_directory /var/crash;
# For PostgreSQL# postgresql.confkernel.core_pattern = /var/lib/postgresql/%u/core.%ppostgresql.conf: kernel.core_pattern
# For Java applications# Heap dumps vs core dumps# -XX:+HeapDumpOnOutOfMemoryError# -XX:HeapDumpPath=/var/log/heapdump.hprof
# For Python# ulimit must be set before starting# Or use: faulthandler modulepython -c "import faulthandler; faulthandler.enable()"89.2 Analyzing Core Dumps
Section titled “89.2 Analyzing Core Dumps”Using GDB
Section titled “Using GDB”# ============================================================# ANALYZING CORE DUMPS WITH GDB# ============================================================
# Load core dump with executablegdb ./program core.1234
# Or specify both separatelygdb -c core.1234 ./program
# Essential GDB Commands:
# Backtracebt # Quick backtracebt full # Backtrace with local variablesbt 20 # Show 20 framesthread apply all bt # All threads
# Frame navigationframe 3 # Switch to frame 3up 2 # Go up 2 framesdown 1 # Go down 1 frameinfo frame # Current frame info
# Variables and memoryprint variable_nameprint *pointerprint array[0]@10 # Print 10 elementsx/100x &buffer # Examine 100 words hexx/s &string # Examine stringx/i &function # Examine at address
# Registersinfo registersinfo all-registers
# Threadsinfo threadsthread apply all btthread 2bt
# Source codelist # Show source around PClist function_namelist 10,20 # Lines 10-20
# Breakpointsinfo breakpointsdelete 1
# Continuing after crash (for debugging)set disassemble-next-line onrun
# Useful commandsinfo signals # Signal handlinginfo proc # Process infoinfo files # Loaded filesPractical Analysis Examples
Section titled “Practical Analysis Examples”# ============================================================# PRACTICAL CORE DUMP ANALYSIS# ============================================================
# Example: Debugging a crashgdb ./myapp core.5678
# 1. Quick overview(gdb) bt#0 0x00007f8a12345678 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
# 2. Find the crash location(gdb) info registers rip#rip 0x555555554a2a <main+42>
# 3. Examine variables at crash(gdb) info locals#i = 10#buffer = 0x0
# 4. Check the backtrace fully(gdb) bt full
# 5. Find null pointer(gdb) print buffer#$1 = 0x0
# 6. Go up to see where buffer was set(gdb) up(gdb) list
# Common crash patterns:
# Null pointer dereference# Program received signal SIGSEGV, Segmentation fault.# 0x0000000000000000 in ?? ()
# Buffer overflow# Program received signal SIGABRT, Aborted.# __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
# Double free# *** glibc detected *** ./program: double free or corruption
# Use after free# Invalid free() / delete / delete[]
# Stack overflow# Program received signal SIGSEGV, Segmentation fault.# 0x00007f8a1c00000 in ?? ()Using crash Utility
Section titled “Using crash Utility”# ============================================================# ANALYZING KERNEL CORE DUMPS WITH CRASH# ============================================================
# Install crash and kernel debug infosudo apt install crashsudo debuginfo-install kernel
# Analyze kernel crash dumpsudo crash vmlinux /var/crash/vmcore
# Common crash commands:crash> ps # List processescrash> bt # Backtracecrash> bt -a # All CPU backtracescrash> kmem -i # Memory infocrash> log # Kernel log buffercrash> files # Open filescrash> net # Network socketscrash> waitq # Wait queuescrash> struct task_struct # Task structurecrash> p <PID> # Process details
# Analyze specificcrash> ps | grep <name> # Find processcrash> rd <address> 20 # Read 20 longs from addrcrash> dis <address> # Disassemble
# Extensionscrash> mod # Kernel modulescrash> irq # IRQ handlers89.3 Production Best Practices
Section titled “89.3 Production Best Practices”Configuration Checklist
Section titled “Configuration Checklist”# ============================================================# PRODUCTION CORE DUMP CONFIGURATION CHECKLIST# ============================================================
# 1. System Configuration# Enable core dumps system-wide# /etc/security/limits.conf* soft core unlimited* hard core unlimited
# /etc/sysctl.confkernel.core_pattern = /var/crash/core.%e.%p.%tkernel.core_uses_pid = 1
# 2. Storage Locationmkdir -p /var/crashchmod 1777 /var/crash
# 3. Log Rotation# /etc/logrotate.d/core dumps/var/crash/core.* { daily rotate 5 compress delaycompress missingok notifempty}
# 4. Application Configuration# Set ulimit in systemd service[Service]LimitCORE=infinity
# 5. Monitoring# Alert if core dumps appearfind /var/crash -type f -mtime -1 -ls
# 6. Security# Restrict core dump location permissions# Prevent information disclosureCore Dump Analysis Workflow
Section titled “Core Dump Analysis Workflow”┌─────────────────────────────────────────────────────────────────────────┐│ CORE DUMP ANALYSIS WORKFLOW │├─────────────────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Step 1: Collect │ ││ │ - Locate core dump file │ ││ │ - Get matching binary/executable │ ││ │ - Gather related logs │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Step 2: Initial Analysis │ ││ │ - Load in gdb: gdb ./program core │ ││ │ - Get backtrace: bt │ ││ │ - Identify crash location │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Step 3: Deep Dive │ ││ │ - Examine variables │ ││ │ - Check register values │ ││ │ - Analyze call chain │ ││ │ - Identify root cause │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │ ││ ▼ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Step 4: Fix and Prevent │ ││ │ - Implement fix │ ││ │ - Add tests │ ││ │ - Add monitoring │ ││ │ - Update code │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘89.4 Interview Questions
Section titled “89.4 Interview Questions”┌─────────────────────────────────────────────────────────────────────────┐│ CORE DUMP INTERVIEW QUESTIONS │├─────────────────────────────────────────────────────────────────────────┤ │Q1: What is a core dump? │ │A1: │A core dump is a file containing: │- Complete memory snapshot of a process at crash time │- CPU register values │- Stack traces for all threads │- Information about loaded shared libraries │- Used for post-mortem debugging of crashes │ │─────────────────────────────────────────────────────────────────────────┤ │Q2: How do you enable core dumps in Linux? │ │A2: │1. System: /etc/security/limits.conf - set core unlimited │2. Kernel: sysctl kernel.core_pattern │3. Process: ulimit -c unlimited in shell │4. Program: setrlimit(RLIMIT_CORE, ...) in code │5. Systemd: LimitCORE=infinity in service file │ │─────────────────────────────────────────────────────────────────────────┤ │Q3: What does kernel.core_pattern mean? │ │A3: │Defines where and how core dumps are saved: │- %p = PID │- %e = executable name │- %t = timestamp │- %h = hostname │- /var/crash/ = directory (if starts with /) │ │─────────────────────────────────────────────────────────────────────────┤ │Q4: How do you analyze a core dump with gdb? │ │A4: │gdb ./program core.1234 │- bt: backtrace │- bt full: backtrace with local variables │- frame N: switch to frame N │- print var: print variable │- x/100x addr: examine memory │- info registers: register values │- thread apply all bt: all threads │ │─────────────────────────────────────────────────────────────────────────┤ │Q5: What are the differences between process and kernel core dumps? │ │A5: │- Process core dump: Single process memory only │- Kernel core dump (vmcore): Entire kernel memory + all processes │- Kernel dumps need special tools (crash, kdump) │- Kernel crashes handled by kdump/kexec │ │─────────────────────────────────────────────────────────────────────────┤ │Q6: How do you prevent core dumps in production for security? │ │A6: │- Set core dump size to 0 │- Use prctl(PR_SET_DUMPABLE, 0) for sensitive processes │- Store core dumps in secure location with restricted permissions │- Consider coredumpctl with systemd │- Avoid including sensitive data in core dumps │ │─────────────────────────────────────────────────────────────────────────┤ │Q7: What is the difference between gcore and gdb -c? │ │A7: │- gdb -c: Analyzes existing core dump │- gcore: Creates core dump of running process │- gcore <PID>: Generate core without killing process │- Useful for debugging live processes or hung programs │ │─────────────────────────────────────────────────────────────────────────┤ │Q8: How do you debug a core dump without debug symbols? │ │A8: │- Rebuild with debug symbols (-g flag) │- Use separate debug symbol packages │- Install debuginfo packages (Fedora/RHEL) │- Use objdump for basic analysis │- Function names may be missing but addresses still useful │ │─────────────────────────────────────────────────────────────────────────┤ │Q9: What is systemd-coredump and how does it work? │ │A9: │- systemd-coredump is systemd's core dump handling │- Catches core dumps via kernel.core_pattern │- Stores in /var/lib/systemd/coredump/ │- Uses coredumpctl to list and extract │- Can limit size, compress, store metadata │- Configure in /etc/systemd/coredump.conf │ │─────────────────────────────────────────────────────────────────────────┤ │Q10: How do you debug multi-threaded core dumps? │ │A10: │- thread apply all bt: backtrace for all threads │- info threads: list all threads │- thread N: switch to thread N │- Use bt full to see all thread-local variables │- Look for deadlock patterns in backtraces │- Check which thread received signal │ │└─────────────────────────────────────────────────────────────────────────┘Quick Reference
Section titled “Quick Reference”# Enable core dumpsulimit -c unlimitedsysctl kernel.core_pattern=/var/crash/core.%p
# Generate core dump of running processgcore <PID>
# Analyze core dumpgdb ./program core.1234
# Common gdb commandsbt # Backtracebt full # Full backtraceframe N # Switch frameprint var # Print variablex/100x &buffer # Examine memory
# Kernel crash analysissudo crash vmlinux vmcorecrash> btcrash> ps
# List core dumpscoredumpctl listSummary
Section titled “Summary”- Core Dump: Memory snapshot of crashed process
- Enable: ulimit, sysctl, systemd service configuration
- Analyze: gdb for user space, crash for kernel
- Commands: bt, print, x/, frame, thread
- Production: Configure storage, rotation, security
Next Chapter
Section titled “Next Chapter”Chapter 90: Hardware Diagnostics
Last Updated: February 2026