Sync_issues

Chapter 46: Sync Problems & Solutions

Overview

Synchronization issues are among the most common problems encountered when running blockchain nodes. A node that fails to sync properly cannot participate in the network, serve RPC requests, or validate transactions. Understanding the various sync issues, their symptoms, and solutions is essential for maintaining a healthy blockchain infrastructure. This chapter provides comprehensive troubleshooting guidance for Ethereum and other blockchain networks.

---

46.1 Understanding Blockchain Synchronization

How Sync Works

┌─────────────────────────────────────────────────────────────┐
│              BLOCKCHAIN SYNC PROCESS                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    │
│  │   Network   │───►│  Discovery  │───►│   Download  │    │
│  │   Peers     │    │   Blocks    │    │   Headers   │    │
│  └─────────────┘    └─────────────┘    └─────────────┘    │
│                                             │               │
│                                             ▼               │
│                                      ┌─────────────┐        │
│                                      │   Execute   │        │
│                                      │   State     │        │
│                                      └─────────────┘        │
│                                             │               │
│                                             ▼               │
│                                      ┌─────────────┐        │
│                                      │   Verify    │        │
│                                      │   Results   │        │
│                                      └─────────────┘        │
│                                             │               │
│                                             ▼               │
│                                      ┌─────────────┐        │
│                                      │  Finalized  │        │
│                                      │    State    │        │
│                                      └─────────────┘        │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Sync Stages

Stage	Description	Duration
Header Sync	Download block headers	Minutes to hours
Body Sync	Download block bodies	Hours
State Sync	Download state trie	Hours to days
Catchup	Near HEAD, executing	Minutes

---

46.2 Stuck Sync

Symptoms

┌─────────────────────────────────────────┐
│         STUCK SYNC INDICATORS             │
├─────────────────────────────────────────┤
│                                          │
│  ✓ Block height not progressing          │
│  ✓ Peer count appears healthy            │
│  ✓ No error messages in logs             │
│  ✓ CPU usage normal                       │
│  ✓ Memory usage normal                    │
│                                          │
│  Example:                                │
│  Current Block: 18500000                 │
│  Target Block:   18500100                │
│  (No change for 30+ minutes)            │
│                                          │
└─────────────────────────────────────────┘

Diagnosis Commands

# Check current block
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Check sync status
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

# Check peer count
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}'

# View recent logs
journalctl -u geth --since "30 minutes ago" | tail -50

Root Causes

Cause	Description	Probability
Database corruption	LevelDB/RocksDB corruption	High
Peer issues	Connected to stale peers	Medium
Consensus issues	Fork in chain	Low
Resource constraints	Memory/disk pressure	Medium
Network partition	Isolated from network	Low

Solutions

1. Fresh Sync (Recommended)

# Stop the node
sudo systemctl stop geth

# Backup important data (keystore only)
cp -r /data/ethereum/keystore /backup/keystore

# Remove chaindata
rm -rf /data/ethereum/geth/chaindata

# Remove state data
rm -rf /data/ethereum/geth/triecache

# Optional: Remove snapshots if corrupted
rm -rf /data/ethereum/geth/snapshots

# Start the node
sudo systemctl start geth

# Monitor sync progress
geth attach http://localhost:8545
> eth.syncing

2. Try Different Bootnodes

# Ethereum Mainnet Bootnodes
geth --bootnodes \
    "enr:-KG4QOtcLhT1LioJW5XHmhLGr9jnoJ5XF8J8p TzW7yGrqDzoP3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8=-BMgBFYHr7tJ3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8@bootnode1.mainnet.ethdisco.net:30303,\
     enr:-Ly4QFn-6sJ8tJ3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8=-BMgBFYHr7tJ3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8@bootnode2.mainnet.ethdisco.net:30303,\
     enr:-Ku4QO7sJ8tJ3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8=-BMgBFYHr7tJ3z6E1T5C9LwK3uK8Q6G7F9B2M1K3W8@bootnode3.mainnet.ethdisco.net:30303"

3. Check System Resources

# Check memory
free -h

# Check disk space
df -h

# Check disk I/O
iostat -x 5

# Check CPU
top
htop

# Check for OOM killer
dmesg | grep -i "out of memory"

4. Reset Database

# Use geth removedb utility
geth removedb --datadir /data/ethereum

# Or specify individual databases
geth removedb --datadir /data/ethereum --chaindata
geth removedb --datadir /data/ethereum --ancient

---

46.3 Slow Sync

Symptoms

┌─────────────────────────────────────────┐
│       SLOW SYNC INDICATORS                │
├─────────────────────────────────────────┤
│                                          │
│  ✓ Sync progressing but very slowly     │
│  ✓ Block height increases slowly        │
│  ✓ High resource usage                  │
│  ✓ Many peer disconnections             │
│                                          │
│  Example Progress:                      │
│  Before: 100 blocks/hour                │
│  After:  10 blocks/hour                 │
│                                          │
└─────────────────────────────────────────┘

Performance Comparison

Sync Mode	Speed	Resource Usage
Full Sync	Slowest	High
Fast Sync	Medium	Medium
Snap Sync	Fast	Medium
Light access	Fastest	Low — but geth removed --syncmode light (v1.13); use dedicated light clients

Solutions

1. Increase Peer Count

# Increase max peers
geth --maxpeers 100 --datadir /data/ethereum

# Check current peer count
geth attach http://localhost:8545
> net.peerCount

2. Optimize Cache

# Increase cache for 32GB+ RAM system
geth --cache 8192 --datadir /data/ethereum

# For 64GB+ system
geth --cache 16384 --datadir /data/ethereum

# Check current cache usage
geth attach http://localhost:8545
> debug.getBlockChainInfo()

3. Use Faster Storage

# Use NVMe instead of SATA SSD
# Check current storage type
lsblk -o NAME,TYPE,SIZE,MODEL

# Benchmark storage
fio --name=seqread --readonly --filename=/tmp/fiotest --ioengine=libaio --iodepth=1 --bs=4M --direct=1 --size=1G --numjobs=1 --runtime=60 --time_based

# Recommended: Samsung 990 Pro, WD Black SN850X

4. Switch to Snap Sync

# Use snap sync (default in recent Geth versions)
geth --syncmode snap --datadir /data/ethereum

# Verify sync mode
geth attach http://localhost:8545
> eth.syncing

---

46.4 No Progress (Not Syncing)

Diagnosis

# 1. Check if actually syncing
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

# Response should show:
# - currentBlock
# - highestBlock
# - knownStates
# - pulledStates

# If not syncing, response is "false"

# 2. Check network connectivity
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"net_listening","params":[],"id":1}'

# 3. Check peers
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}'

Solutions

1. Verify Network Connection

# Test internet connectivity
ping -c 4 8.8.8.8
ping -c 4 google.com

# Test specific ports
nc -zv bootnode.mainnet.ethdisco.net 30303

# Check firewall
sudo ufw status
sudo iptables -L -n

2. Check Logs for Errors

# View recent errors
journalctl -u geth --since "1 hour ago" | grep -i "error"

# Check for specific issues
journalctl -u geth | grep -i "no peers"
journalctl -u geth | grep -i "connection refused"
journalctl -u geth | grep -i "timeout"

3. Restart with Fresh State

# Stop node
sudo systemctl stop geth

# Backup state (optional)
cp -r /data/ethereum/geth/chaindata /backup/chaindata-backup

# Remove state
rm -rf /data/ethereum/geth/chaindata
rm -rf /data/ethereum/geth/triecache

# Restart with new peers
sudo systemctl start geth

---

46.5 Fork Issues

Detecting Forks

# Check if you're on a fork
# Compare your block with external block explorers

# Get your current block
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Compare with:
# - https://etherscan.io/blocks
# - https://blockchain.info/

# Check for reorgs
geth attach http://localhost:8545
> debug.traceBlockByNumber(eth.blockNumber)

Resolving Fork Issues

# 1. Check which fork you're on
geth attach http://localhost:8545
> eth.getBlock(eth.blockNumber).hash

# 2. Compare with canonical chain
# Etherscan API
curl "https://api.etherscan.io/api?module=block&action=getblocknobytime&timestamp=$(date +%s)&closest=before"

# 3. If on wrong fork, resync
sudo systemctl stop geth
rm -rf /data/ethereum/geth/chaindata
sudo systemctl start geth

---

46.6 Era Merge / History Issues

Understanding Erigon/Post-Merge Issues

After The Merge, syncing became more complex:

┌─────────────────────────────────────────────────────────────┐
│           POST-MERGE SYNC CHALLENGES                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Execution Layer (EL) ◄──► Consensus Layer (CL)              │
│                                                              │
│  Issues:                                                    │
│  ✓ CL node must be running for EL to sync                  │
│  ✓ EL needs authenticated connection to CL                 │
│  ✓ Different sync modes for EL vs CL                       │
│  ✓ Beacon chain data required                               │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Fix EL/CL Sync Issues

# 1. Ensure both EL and CL are running
systemctl status geth
systemctl status lighthouse
systemctl status prysm
systemctl status teku

# 2. Check EL/CL connection — Engine API is JWT-authenticated on port 8551,
#    NOT exposed on the public HTTP API, and there is no
#    "debug.getExecutionEnginePayload" method. Real Engine API calls:
curl -s -X POST http://localhost:8551 \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(cat /data/ethereum/geth/jwtsecret)" \
    -d '{"jsonrpc":"2.0","method":"engine_getPayloadV3","params":["0x0","0x0"],"id":1}'
# (a valid JWT + any response = EL/CL connection works)

# 3. Check JWT secret
cat /data/ethereum/geth/jwtsecret
# Should match CL configuration

# 4. If using Lighthouse (note: --execution-jwt-secret, not --execution-jwt):
lighthouse --network mainnet \
    --execution-endpoint http://localhost:8551 \
    --execution-jwt-secret /data/ethereum/geth/jwtsecret

# 5. Check for sync progress
curl -s -X GET http://localhost:5052/eth/v1/node/syncing | jq

---

46.7 Erigon-Specific Sync Issues

Erigon Common Problems

# Erigon requires specific flags
erigon \
    --datadir /data/ethereum \
    --chain mainnet \
    --http \
    --http.addr 0.0.0.0 \
    --http.port 8545 \
    --http.api eth,net,debug,trace,txpool \
    --ws

# Check Erigon sync status
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

# Erigon flags for sync issues (there is no --prune.mode flag; pruning is
# configured with --prune, and "--prune.archive" does not exist — omitting
# --prune keeps everything, i.e. archive behaviour):
# --prune htc              # prune history, tx-index, call traces
# --externalcl             # use an external consensus client (real flag)

---

46.8 Validator Sync Issues

Lighthouse/Prysm Specific

# Check Lighthouse sync status
curl -X GET http://localhost:5052/eth/v1/node/syncing | jq

# Check block proposed
curl -X GET http://localhost:5052/eth/v1/beacon/headers | jq

# Check validator duties
curl -X GET http://localhost:5052/eth/v1/validator/duties/proposer/1200 | jq

# Check if beacon chain synced
curl -X GET http://localhost:5052/eth/v1/node/health | jq

Fix Validator Sync

# 1. Check if beacon chain is synced (Lighthouse's "bn" subcommand accepts
#    --network mainnet — there is no --testnet flag and no "status" subcommand)
curl -s -X GET http://localhost:5052/eth/v1/node/syncing | jq
# → "is_syncing": false means synced

# 2. Check EL connection
curl -X POST http://localhost:8551 \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(cat /data/ethereum/geth/jwtsecret)" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# 3. Reset beacon chain data (stop first; Lighthouse has no "bn db clear" command —
#    rename the beacon data dir to resync cleanly, keeping the old copy as backup)
systemctl stop lighthouse
sudo mv /var/lib/lighthouse/mainnet /var/lib/lighthouse/mainnet.old
systemctl start lighthouse

# 4. Restart
systemctl restart lighthouse

---

46.9 Network-Specific Issues

Testnet vs Mainnet

# Testnet Sync (Sepolia)
geth --sepolia \
    --syncmode snap \
    --bootnodes "enr:-KG4QO..."

# Testnet Sync (Goerli - deprecated)
geth --goerli \
    --syncmode snap

# Mainnet Sync
geth --mainnet \
    --syncmode snap

Private Network Issues

# Check genesis file matches network
cat /data/ethereum/geth/genesis.json | jq '.config'

# Verify network ID
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"net_version","params":[],"id":1}'

# Check if bootnodes are correct
geth --bootnodes "enr:..." --networkid 12345

---

46.10 Monitoring & Prevention

Prometheus Alerts

prometheus/alerting_rules.yml

groups:
- name: sync_alerts
  rules:
  - alert: NodeNotSyncing
    expr: increase(chain_head_block{job="geth"}[10m]) == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Node {{ $labels.instance }} is not syncing"

  - alert: SyncStalled
    expr: rate(chain_head_block[5m]) == 0
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.instance }} sync stalled"

  - alert: SyncTooSlow
    expr: rate(chain_head_block[5m]) < 10
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.instance }} sync too slow"

  - alert: NodeBehindNetwork
    # No "ethereum_network_block" metric exists. Two real options:
    # (a) compare import head vs the highest header the node has seen:
    expr: (chain_head_header - chain_head_block) > 100
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.instance }} is behind by 100+ blocks"
    # (b) for tip-from-external-source alerting (textfile collector), see ch38

Health Check Script

check_sync.sh

#!/bin/bash
RPC_URL="http://localhost:8545"
MAX_BLOCK_DIFF=10

# Get current block (hex → decimal conversion; "0x$(cat)" piped into printf
# is broken — use jq's tonumber instead)
CURRENT=$(curl -s -X POST $RPC_URL \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    | jq -r '.result | tonumber')

# Get network block (from block explorers)
NETWORK_BLOCK=$(curl -s "https://api.etherscan.io/api?module=block&action=getblocknobytime&timestamp=$(date +%s)&closest=before" \
    | jq -r '.result[0].blockNumber')

DIFF=$((NETWORK_BLOCK - CURRENT))

if [ $DIFF -gt $MAX_BLOCK_DIFF ]; then
    echo "WARNING: Node is $DIFF blocks behind network"
    exit 1
else
    echo "OK: Node is synced (diff: $DIFF)"
    exit 0
fi

---

46.11 Quick Reference

Common Commands

Issue	Solution
Stuck at block X	rm -rf /data/ethereum/geth/chaindata && systemctl restart geth
No peers	Add bootnodes, check firewall
Slow sync	Increase peers, use snap sync, upgrade hardware
Database corruption	geth removedb then resync
Fork issues	Compare with Etherscan, resync if needed
Post-Merge issues	Ensure CL node running, check JWT

Emergency Commands

# Nuclear option - delete everything except keys
systemctl stop geth
rm -rf /data/ethereum/geth/*
cp -r /backup/keystore /data/ethereum/keystore
systemctl start geth

---

Summary

• Stuck sync is often solved by fresh sync or database reset
• Slow sync requires hardware upgrades or configuration tuning
• No progress is usually network or resource related
• Post-Merge requires both EL and CL to be synced
• Monitoring is critical for early problem detection
• Prevention: Regular health checks, adequate resources, updated software

---

Next Chapter

In Chapter 47: Memory & CPU Optimization, we’ll explore performance optimization techniques.

---

Last Updated: 2026-02-22