Upgrades

Chapter 41: Node Software Upgrades

Overview

Regular software upgrades are essential for maintaining blockchain node security, performance, and network compatibility. Blockchain protocols frequently undergo hard forks, protocol upgrades, and security patches that require node operators to update their software. This chapter provides comprehensive guidance on planning, executing, and verifying node upgrades across different blockchain clients.

---

41.1 Upgrade Planning

Upgrade Checklist

┌─────────────────────────────────────────────────────────────┐
│                 UPGRADE PLANNING CHECKLIST                     │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Pre-Upgrade:                                               │
│  ☐ Review release notes for new version                   │
│  ☐ Check for hard fork requirements                        │
│  ☐ Verify hardware compatibility                            │
│  ☐ Test on testnet/staging first                          │
│  ☐ Schedule maintenance window                             │
│  ☐ Notify users/stakers                                    │
│  ☐ Create full backup                                      │
│  ☐ Document current configuration                          │
│                                                              │
│  During Upgrade:                                            │
│  ☐ Stop node gracefully                                    │
│  ☐ Backup database (optional incremental)                  │
│  ☐ Install new binary                                      │
│  ☐ Verify binary checksum                                  │
│  ☐ Update configuration if needed                          │
│  ☐ Start node                                              │
│  ☐ Verify sync and health                                  │
│                                                              │
│  Post-Upgrade:                                              │
│  ☐ Monitor logs for errors                                  │
│  ☐ Verify block sync                                       │
│  ☐ Check peer connections                                   │
│  ☐ Confirm RPC functionality                               │
│  ☐ Update monitoring/dashboards                            │
│  ☐ Document any issues                                    │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Understanding Upgrade Types

Type	Frequency	Risk	Downtime
Patch	Monthly	Low	Minimal
Minor	Quarterly	Medium	Short
Major	Yearly	High	Extended
Hard Fork	Variable	High	Coordinated

---

41.2 Manual Upgrade Process

Ethereum (Geth)

# Step 1: Check current version
geth version

# Step 2: Stop the node gracefully
sudo systemctl stop geth

# Step 3: Wait for graceful shutdown
# Node should finish current operations
sleep 30

# Step 4: Verify node is stopped
systemctl status geth

# Step 5: Download new version
# Option A: From official releases
wget https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.13.14-8ca808b2.tar.gz

# Option B: From GitHub
wget https://github.com/ethereum/go-ethereum/releases/download/v1.13.14/geth-linux-amd64-1.13.14-8ca808b2.tar.gz

# Step 6: Verify checksum
sha256sum geth-linux-amd64-1.13.14-8ca808b2.tar.gz
# Compare with checksums.txt from release page

# Step 7: Extract
tar -xzf geth-linux-amd64-1.13.14-8ca808b2.tar.gz

# Step 8: Install new binary
sudo mv geth /usr/local/bin/

# Step 9: Verify installation
geth version

# Step 10: Start the node
sudo systemctl start geth

# Step 11: Monitor logs
journalctl -fu geth

# Step 12: Verify sync
geth attach http://localhost:8545
> eth.blockNumber
> eth.syncing

Erigon

# Erigon upgrade
# Erigon is typically run from docker or built from source

# Using Docker
docker pull thorax/erigon:v2.58.0
docker stop erigon
docker rm erigon
docker run -d \
    --name erigon \
    -v erigon-data:/home/erigon/.local/share/erigon \
    -p 30303:30303 \
    -p 30303:30303/udp \
    -p 8545:8545 \
    -p 8546:8546 \
    thorax/erigon:v2.58.0

# From source (if built locally)
cd erigon
git fetch
git checkout v2.58.0
make erigon
sudo systemctl stop erigon
make run_erigon

Nethermind

# Nethermind upgrade
wget https://github.com/NethermindEth/nethermind/releases/download/1.25.0/nethermind-linux-x64-1.25.0-9bccabfe.tar.gz
tar -xzf nethermind-linux-x64-1.25.0-9bccabfe.tar.gz

# Stop current node
sudo systemctl stop nethermind

# Replace binary
sudo cp -r nethermind/* /opt/nethermind/
sudo systemctl start nethermind

---

41.3 Using Cosmovisor (Cosmos)

Cosmovisor is the recommended upgrade mechanism for Cosmos SDK chains.

Installation

# Install cosmovisor
cd /tmp
wget https://github.com/cosmos/cosmos-sdk/releases/download/v0.47.0/cosmovisor-v0.47.0-linux-amd64.tar.gz
tar -xzf cosmovisor-v0.47.0-linux-amd64.tar.gz
sudo mv cosmovisor /usr/local/bin/

# Verify
cosmovisor version

Configuration

# Set environment variables
export DAEMON_NAME=gaiad
export DAEMON_HOME=~/.gaia
export DAEMON_ALLOW_DOWNLOAD_BINARIES=true
export DAEMON_RESTART_AFTER_UPGRADE=true

# Create directory structure
mkdir -p $DAEMON_HOME/cosmovisor/upgrades
mkdir -p $DAEMON_HOME/cosmovisor/genesis/bin

# Copy initial binary
sudo cp $(which gaiad) $DAEMON_HOME/cosmovisor/genesis/bin/

# Or use systemctl with environment
sudo systemctl edit cosmovisor

Systemd Service

/etc/systemd/system/cosmovisor.service

[Unit]
Description=Cosmos Validator Cosmovisor
After=network-online.target

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu
Environment="DAEMON_NAME=gaiad"
Environment="DAEMON_HOME=/home/ubuntu/.gaia"
Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=true"
Environment="DAEMON_RESTART_AFTER_UPGRADE=true"
ExecStart=/usr/local/bin/cosmovisor run start
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Preparing Upgrades

# Create upgrade directory structure
export DAEMON_HOME=~/.gaia

# For upgrade "v1.5.0"
mkdir -p $DAEMON_HOME/cosmovisor/upgrades/v1.5.0/bin

# Copy new binary
cp gaiad $DAEMON_HOME/cosmovisor/upgrades/v1.5.0/bin/

# Cosmovisor will automatically switch to new binary at upgrade height

---

41.4 Lighthouse (Ethereum 2)

Upgrade Process

# Check current version
lighthouse --version

# Stop the node
sudo systemctl stop lighthouse

# Update binary
cd /tmp
wget https://github.com/sigp/lighthouse/releases/download/v4.5.0/lighthouse-v4.5.0-x86_64-unknown-linux-gnu.tar.gz
tar -xzf lighthouse-v4.5.0-x86_64-unknown-linux-gnu.tar.gz
sudo cp lighthouse /usr/local/bin/

# Verify
lighthouse --version

# Start
sudo systemctl start lighthouse

# Check logs
journalctl -fu lighthouse

Automatic Updates with Lighthouse

# Lighthouse can auto-update usingBeaconChain binary
lighthouse bn \
    --checkpoint-sync-url https://checkpoint.alphavantage.io \
    --fallback-checkpoint-urls https://beaconstate.info \
    --http \
    --enable-private-discovery \
    --slasher

---

41.5 Validator-Specific Upgrades

Ethereum Validator Upgrades

# Upgrade validator keys (if needed)
# Always backup before upgrading

# Check validator status
curl -X GET http://localhost:5052/eth/v1/node/syncing

# Check if beacon chain is synced
curl -X GET http://localhost:5052/eth/v1/node/health

# After upgrade, verify duties
curl -X GET http://localhost:5052/eth/v1/validator/duties/attester/$(date +%s)

Validator Upgrade Checklist

┌─────────────────────────────────────────────────────────────┐
│              VALIDATOR UPGRADE CHECKLIST                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ☐ Coordinate with other validators in network              │
│  ☐ Check upgrade proposal timing                            │
│  ☐ Verify validator keys work with new version              │
│  ☐ Have fallback (testnet validator)                        │
│  ☐ Prepare emergency response plan                          │
│  ☐ Schedule during low-activity period                      │
│  ☐ Monitor after upgrade                                     │
│  ☐ Check attestation effectiveness                          │
│                                                              │
└─────────────────────────────────────────────────────────────┘

---

41.6 Hard Fork Upgrades

Preparing for Hard Forks

# Step 1: Monitor for fork announcements
# Ethereum: https://ethereum.org/en/history/

# Step 2: Get upgrade details
# - Fork name
# - Block height
# - Required client versions
# - Any configuration changes

# Step 3: Test on testnet
# Use Goerli/Sepolia first

# Step 4: Prepare configuration
# Update genesis file if needed
# Update bootnodes if provided
# Configure any new flags

Hard Fork Script

upgrade_hardfork.sh

#!/bin/bash
FORK_BLOCK=19426587
CLIENT_VERSION="1.13.14"

echo "=== Hard Fork Upgrade Script ==="
echo "Fork Block: $FORK_BLOCK"
echo "Required Version: $CLIENT_VERSION"

# Check current block
CURRENT_BLOCK=$(curl -s -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
    | jq -r '.result' | printf "%d\n" "0x$(cat)")

BLOCKS_UNTIL_FORK=$((FORK_BLOCK - CURRENT_BLOCK))

echo "Current Block: $CURRENT_BLOCK"
echo "Blocks Until Fork: $BLOCKS_UNTIL_FORK"

# Calculate estimated time (12 seconds per block)
HOURS=$((BLOCKS_UNTIL_FORK * 12 / 3600))
echo "Estimated time until fork: ~$HOURS hours"

# Check if client version is compatible
CURRENT_VERSION=$(geth version | grep -oP 'Version: \K[0-9.]+')
echo "Current Client Version: $CURRENT_VERSION"

if [ "$CURRENT_VERSION" = "$CLIENT_VERSION" ]; then
    echo "✓ Client version is compatible"
else
    echo "⚠ Client upgrade required before fork!"
fi

---

41.7 Best Practices

Testing Upgrades

# Always test on testnet first
# Example: Goerli (deprecated) or Sepolia

# Start testnet node
geth --sepolia --syncmode snap --http

# After upgrade, verify:
# 1. Sync status
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}'

# 2. Block production
sleep 60
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# 3. RPC calls
curl -X POST http://localhost:8545 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}'

Upgrade Rollback Plan

rollback.sh

#!/bin/bash
# Keep previous version binary
sudo cp /usr/local/bin/geth /usr/local/bin/geth-backup

# To rollback:
sudo systemctl stop geth
sudo mv /usr/local/bin/geth-backup /usr/local/bin/geth
sudo systemctl start geth

Monitoring After Upgrade

# Watch for errors
journalctl -fu geth --since "1 hour ago" | grep -i error

# Check peer connections
geth attach http://localhost:8545
> net.peerCount

# Check sync
> eth.syncing
> eth.blockNumber

# Monitor performance
> debug.getBlockChainInfo()

---

41.8 Troubleshooting Upgrade Issues

Common Problems

Issue	Cause	Solution
Node won’t start	Configuration mismatch	Review logs, check flags
Sync stuck	Database format change	Fresh sync required
Peers not connecting	New P2P protocol	Update bootnodes
RPC errors	API changes	Update client code

Recovery Commands

# Full resync after failed upgrade
sudo systemctl stop geth
rm -rf /data/ethereum/geth/chaindata
sudo systemctl start geth

# Use different sync mode
geth --syncmode snap --http

# Check logs for specific errors
journalctl -u geth | grep -i "panic"
journalctl -u geth | grep -i "fatal"

---

Summary

• Plan upgrades carefully with checklists and testing
• Test on testnet before production deployment
• Backup everything before upgrading
• Use Cosmovisor for automated Cosmos chain upgrades
• Monitor closely after any upgrade
• Have rollback plan ready
• Coordinate with network for hard forks
• Keep clients updated to stay compatible

---

Next Chapter

In Chapter 42: Backup & Recovery, we’ll explore data backup strategies.

---

Last Updated: 2026-02-22