Debugging

Chapter 49: Debugging Tools & Techniques

Overview

This chapter covers tools and techniques for debugging blockchain nodes: CPU/memory profiling, the debug RPC namespace, transaction tracing, and where to look when something subtle is wrong.

---

49.1 pprof Profiling

Enable pprof

# Real geth flag names: --pprof.addr / --pprof.port (dotted, like the metrics flags)
geth --pprof --pprof.addr 0.0.0.0 --pprof.port 6061

⚠️ Bind pprof to localhost in production (--pprof.addr 127.0.0.1) — the pprof HTTP endpoint leaks process internals and lets anyone trigger CPU profiles.

Access

# View profiles
curl http://localhost:6061/debug/pprof/

# CPU profile (30s sample, then analyze)
go tool pprof http://localhost:6061/debug/pprof/profile

# Heap profile
go tool pprof http://localhost:6061/debug/pprof/heap

# Goroutine dump (text)
curl -s http://localhost:6061/debug/pprof/goroutine?debug=1

# Interactive — or dump and open:
curl -s -o cpu.prof http://localhost:6061/debug/pprof/profile?seconds=30
go tool pprof -top cpu.prof

What to look for: functions with >50% cumulative time (hot loop, compaction, JSON-RPC marshaling), heap growth without GC (leak), and unmatched goroutines (leaked connections).

---

49.2 Debug API

Enable Debug API

geth --http --http.api eth,net,web3,debug

Common Debug Calls

# Get block as RLP (for inspecting the raw serialized block)
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_getBlockRlp","params":["0x0"],"id":1}'

# Get raw pending tx from the pool
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_getRawTransaction","params":["0x<txhash>"],"id":1}'

# Re-run a block's execution against the current state (fingerprinting)
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["0x10d4f","{}"],"id":1}'

The debug_ namespace is powerful and slow — never expose it on a public RPC endpoint (Chapter 22).

---

49.3 Trace Transactions

Geth tracing

geth exposes traces in the debug_ namespace (debug_traceTransaction, debug_traceCall, debug_traceBlockByNumber); it does not implement the trace_* namespace:

# Trace a single transaction (opcodes + stack + storage deltas)
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_traceTransaction","params":["0x<txhash>","{\"tracer\":\"callTracer\",\"timeout\":\"10s\"}"],"id":1}'

Erigon / Besu / Nethermind tracing

Those clients expose the OpenEthereum-style trace_ namespace:

# Erigon/Besu: trace an entire block
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"trace_block","params":["0x0"],"id":1}'

# trace_replayBlockTransactions — full VM trace for every tx in a block
curl -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"trace_replayBlockTransactions","params":["0x10d4f",["trace"]],"id":1}'

⚠️ trace_block only exists on Erigon/Besu/Nethermind, not on geth. Match the method to the client (a very common interview trap).

---

49.4 System Tools

# Check CPU
top
htop

# Check Memory
free -h

# Check Disk I/O
iostat -x 1

# Check Network
netstat -i
ss -tn state established | wc -l      # open connections

---

49.5 Reading a Crash / Stack Trace

When a client crashes, the first questions are: where did it die and why?

# Full kernel + userspace OOM evidence
dmesg -T | tail -50
journalctl -u geth -n 200 --no-pager

# Core dumps (if enabled)
coredumpctl list geth 2>/dev/null | tail
coredumpctl info geth 2>/dev/null | tail -80

Workflow:

1. Identify the first error line (not later consequences) in the log.
2. Classify: OOM (memory), disk full (I/O), nil/assertion (bug — reproduce + report with --verbosity 5 logs), or panic in a dependency (database corruption).
3. For stack traces: Go clients print the full panic stack to stderr — save it; it pins the module (e.g., eth/downloader, les, trie).
4. If the process hung (not crashed): curl http://localhost:6061/debug/pprof/goroutine?debug=2 shows blocked-on stacks — look for I/O waits vs lock contention.

---

49.6 Debugging Checklist

Symptom	Start here	Then try
Slow RPC	debug/pprof/profile + nginx access log latency	Trace the specific method (debug_traceCall)
Leak (memory grows)	debug/pprof/heap twice, 10 min apart	Look for growth in a single profile → identify caller
Stuck sync	Goroutine dump: which module is blocked	Disk I/O check (ch47), peer check (ch48)
Crash on startup	journalctl first lines after boot	geth db stat (ch44); resync if DB corrupt
Intermittent errors	Enable --verbosity 5 + JSON logs	Reproduce, correlate with metrics (ch36)

---

Summary

• Use pprof (--pprof.addr, --pprof.port) for CPU/memory/goroutine analysis — keep it localhost-only
• debug_ namespace (geth) for raw-block/RPC-level debugging; trace_ namespace is Erigon/Besu/Nethermind
• Always match tracing methods to the client you run
• Crash triage: first error line → classify (OOM/disk/bug/corruption) → reproduce with verbose logs
• A hung node: goroutine dump ?debug=2 shows exactly where it’s blocked

---

Last Updated: 2026-10-07