Common_issues

Chapter 45: Common Node Issues

Overview

This chapter covers the most common issues you’ll encounter when running blockchain nodes. Each issue follows the same diagnostic structure used across Chapters 45–48:

SYMPTOM → CAUSES → COMMANDS → MEANING → ACTION

---

45.1 Sync Issues

Issue A: Node Stuck at Header (Not Importing Blocks)

SYMPTOM:

• eth.syncing returns {currentBlock: X, highestBlock: Y} and currentBlock never advances
• Peer count looks healthy (10+ peers)

CAUSES:

1. State download stuck (snap sync peer that serves state disappeared)
2. Bad peers serving garbage headers
3. Disk I/O saturated (compaction storm)
4. Node time/clock skew

COMMANDS:

# Is it really stuck? Sample twice, 2 minutes apart:
curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
# Also check metrics: increase(chain_head_block[10m]) == 0 → confirmed stalled

# Check disk I/O during the stall:
iostat -x 1

MEANING:

• eth_syncing returning a currentBlock that doesn’t move = the node is mid-sync but stuck fetching state; eth_blockNumber unchanged confirms it.
• High disk %util with low CPU = LSM compaction or sync writing too fast for the disk.

ACTION:

# 1. Let it run 15-30 min — snap state download can pause then resume
# 2. Restart the client (sync is resumable; never delete chaindata first!)
systemctl restart geth
# 3. Only if repeatedly stuck at the SAME height: resync after backing up keys
systemctl stop geth
rm -rf /data/ethereum/geth/chaindata   # ← last resort, costs hours-days
systemctl start geth

⚠️ Deleting chaindata is destructive and slow. Exhaust restart + peer/disk checks first. If you need to keep the node’s history, make a snapshot (Chapter 44) instead.

Issue B: Very Slow Sync

SYMPTOM: Sync is progressing but takes far longer than expected (weeks instead of days).

CAUSES:

1. Disk too slow (SATA SSD vs NVMe) — the #1 cause
2. Too few peers (--maxpeers default 50 is low for initial sync)
3. --cache too small (geth default 1 GB is far too little)
4. Network bandwidth throttling

COMMANDS:

top                      # CPU? single core pegged?
iostat -x 1              # disk util/IOPS
free -h                  # RAM
curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'  # hex peers

MEANING: iostat with high %util while sync rate is low = disk-bound. net_peerCount below ~10 hex=0x0a → peer starvation.

ACTION:

# Right-size cache (use ~50-75% of free RAM) and raise peer limit
geth --syncmode snap --cache 8192 --maxpeers 200
# Move to NVMe if disk-bound (Chapter 47); use snapshots to skip the longest part (Ch 44)

---

45.2 Network Issues

Issue C: No Peers

SYMPTOM: net.peerCount returns 0x0 for hours after startup.

CAUSES:

1. Firewall blocks UDP/TCP 30303
2. Outbound UDP blocked by the host/cloud security group
3. No bootnodes configured / DNS discovery disabled
4. NAT: incoming connections fail behind strict NAT

COMMANDS:

sudo ufw status
# Listeners
ss -lunp | grep 30303 && ss -ltnp | grep 30303
# Try to reach a bootnode
nc -u -vz 3.209.45.79 30303 || true     # example IP; prefer real bootnode IPs
# Ask the node what it knows
geth attach http://localhost:8545 -exec 'admin.peers'
geth attach http://localhost:8545 -exec 'net.peerCount'

MEANING: Stop the command sequence at the first failure: firewall rules blocking the port, ss showing no listener (client not started with --port?), or admin.peers empty (discovery broken).

ACTION:

• Open 30303/tcp and 30303/udp inbound; verify cloud security-group rules (Chapter 25).
• Ensure you run with bootnodes/--discovery.dns for public networks.
• Set a public IP explicitly if behind NAT: geth auto-detects via UPnP; if that fails use --nat=extip:<your-ip>.

Issue D: Peers Dropping / Flapping

SYMPTOM: Peer count fluctuates wildly (20 → 3 → 18 → 2).

CAUSES:

1. Unstable network path (packet loss)
2. Client banning peers aggressively (bad behavior detection)
3. IP-penalty storms from peered with low-quality public peers
4. Node behind NAT without stable port mapping

COMMANDS:

journalctl -u geth --since "1 hour ago" | grep -iE "dial|drop|ban|peer"
# Check for repeated "disconnected" of the same peer

MEANING: Log lines like Dial error/dropping peer for the same IPs over and over = path flapping or a bad peer neighbrhood.

ACTION: Improve network stability; prefer --maxpeers moderate (50-100); rely on persistent/static peers (Chapter 26) for critical connectivity; avoid public-nodes-with-DDoS-peer-flood scenarios. There is no geth --dial.timeout flag — dial behavior is internal.

---

45.3 Performance Issues

Issue E: High CPU Usage After Sync

SYMPTOM: CPU pegged at 100% hours/days after initial sync completed.

CAUSES:

1. RPC abuse (eth_call/eth_getLogs floods — check Nginx/HAProxy logs)
2. LSM compaction running constantly
3. Too-small cache → thrashing
4. debug_/trace_ API exposed publicly

COMMANDS:

top -o %CPU
ss -tn state established | wc -l            # connection count
journalctl -u geth --since "30 min ago" | grep -iE "compaction|write slow"

MEANING: If the top process is the reverse proxy (nginx/haproxy), the load is RPC-driven — check per-method latency (Chapter 22). If geth itself is hot, look for compaction or a too-small cache.

ACTION:

# Right-size the cache upward before reducing anything else
geth --cache 8192
# Remove debug/trace from public HTTP API; rate-limit heavy methods
geth --http.api eth,net,web3

Issue F: Out of Memory / OOM Killed

SYMPTOM: Process disappears, dmesg shows out of memory: killed process, or journalctl -k shows OOM.

CAUSES:

1. --cache set larger than free RAM (caches live in RAM!)
2. Memory growth during long sync (state cache grows)
3. Too-small system (snap sync of mainnet wants 8-16 GB just for caches)
4. No swap configured as a safety valve

COMMANDS:

free -h
dmesg -T | grep -i "killed process"
journalctl -k --since "1 hour ago" | grep -i oom

MEANING: killed process ... total-vm:... line identifies the victim and the memory pressure context.

ACTION:

# Reduce cache to fit RAM with headroom
geth --cache 4096
# Add swap as a crash buffer (still not a substitute for RAM)
sudo fallocate -l 16G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Issue G: Disk Full

SYMPTOM: df -h shows 100%; node falls behind or crashes; no space left on device in logs.

CAUSES:

1. Long --history.state/--history.transactions retention
2. No pruning configured (pruned vs archive)
3. Log files growing unbounded (Chapter 37 rotation missing)
4. Snapshots/backups stored on the same disk

COMMANDS:

df -h
du -sh /data/ethereum/* | sort -h
journalctl -u geth --since "30 min ago" | grep -i "no space"

MEANING: du shows where the bytes live: chaindata (state/blocks), ancient (old blocks/receipts), geth.ipc (tiny), or log rotation is missing.

ACTION:

# Geth: cap history retention (real flags — there is no --pruneancientstore)
geth --history.state 100000 --history.transactions 200000
# Enable log rotation (logrotate) if logs are the culprit
# Prune/archive decisions: see Chapter 30 (Pruning & Data Management)

⚠️ Never rm -rf /data/ethereum/geth/chaindata/ancient to “free space” — it destroys historical data and can corrupt the DB view. Use the documented retention flags or resync intentionally.

---

45.4 RPC Issues

Issue H: RPC Timeouts

SYMPTOM: Client requests time out while eth_blockNumber still answers quickly.

CAUSES:

1. eth_call/eth_getLogs queries doing heavy work (long ranges, complex calls)
2. Proxy timeout shorter than node response time
3. Node overloaded (many huge requests in parallel)
4. --http.timeouts misconfigured (geth default may be too low for heavy calls)

COMMANDS:

# Reproduce with timings:
time curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0x1000"}],"id":1}'
# Check node-side RPC history:
curl -s http://localhost:6060/debug/metrics/prometheus | grep rpc_duration

MEANING: A multi-second time output on a wide eth_getLogs = expected behavior, not a bug — the node is doing real work.

ACTION:

# Nginx: raise read timeout
proxy_read_timeout 300s;
# Or (carefully) raise geth's HTTP timeouts:
geth --http.timeouts read=300,write=300,idle=120
# Better: cap/cache heavy endpoints at the proxy (Chapter 22)

Issue I: RPC Returns Errors

SYMPTOM: Random -32000/-32603 errors, or intermittent failures.

CAUSES:

1. Node not fully synced → evidence: eth_syncing returns an object (still syncing)
2. Client sending bad params (nonce, gas, ABI)
3. Node overloaded → request queue full
4. Upstream gateway rate limiter (reverse proxy / provider) returning 429s

COMMANDS:

# 1. Synced?
curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'
# 2. Healthy?
curl -s -X POST http://localhost:8545 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"net_version","params":[],"id":1}'
# 3. Logs
journalctl -fu geth -n 100

MEANING: eth_syncing returning false = synced; returning an object = still catching up (a classic cause of “errors that come and go”). Rate-limit rejections appear in the reverse-proxy access logs as 429s.

ACTION: Wait for sync to complete; fix bad params at the client; if rate-limited, raise the limit or add capacity (Chapter 22).

---

45.5 Database Issues

Issue J: Database Corruption / Refuses to Start

SYMPTOM: Node crashes on startup, database: file is not a database / corruption errors, or sync fails repeatedly at the same block.

CAUSES:

1. Hard crash/power loss during write (WAL recovery failed)
2. Disk full during a write
3. Filesystem on non-durable mounts (e.g., VM disk removed)
4. Copying chaindata while the node is running

COMMANDS:

journalctl -u geth --since "1 hour ago" | grep -iE "corrupt|leveldb|invalid"
# geth can sometimes repair:
geth db inspect --datadir /data/ethereum   # inventory
# LevelDB-style compaction/repair attempt via remove+resync is the reliable path

MEANING: Look for the cause in the logs (OOM, disk-full, unexpected shutdown) before deciding repair vs resync — fixing only the symptom repeats the corruption.

ACTION:

systemctl stop geth
# Back up the ancient dir if you want to keep history:
mv /data/ethereum/geth/ancient /data/ethereum/geth/ancient.bak
# Remove chaindata and resync (this is the reliable fix; no guaranteed in-place repair)
rm -rf /data/ethereum/geth/chaindata
systemctl start geth
# Investigate the root cause: hw, filesystem, or abrupt termination — add
# RestartSec + clean shutdown procedures (Chapter 41)

---

45.6 Quick Diagnostics

Checklist

# 1. Check if running
systemctl status geth
ps aux | grep geth

# 2. Check logs
journalctl -fu geth -n 50

# 3. Check sync status (object = syncing, false = synced)
curl -s -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

# 4. Check peer count (hex → 0x0a = 10 peers; net.peerCount, not eth.peerCount)
curl -s -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'

# 5. Check block height
curl -s -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

Reading the output: eth_syncing: false + a sensible eth_blockNumber + healthy peers ⇒ node healthy. Any one failing ⇒ work through the matching issue above (45.1–45.5).

---

Summary

• Stuck sync: sample twice, check disk I/O, restart before deleting chaindata (last resort)
• No peers: firewall/UDP/bootnodes/NAT, in that order
• Slow sync: NVMe + --cache + --maxpeers
• OOM/disk full: right-size cache/history retention before anything destructive
• RPC errors: confirm sync completed and check rate limits/proxy
• Database corruption: fix the root cause (power, disk, copy while running), then resync
• Always use net.peerCount/eth_syncing with the correct RPC methods, and real flags (--history.state, --http.timeouts) — never invented ones

---

Next Chapter

In Chapter 46: Sync Problems & Solutions, we’ll deep-dive into sync troubleshooting.

---

Last Updated: 2026-10-07