Metrics

Chapter 36: Node Metrics & Prometheus

Overview

Monitoring your blockchain node is essential for maintaining performance, detecting issues, and ensuring uptime. This chapter lists the real metric names your client exposes, what each one means, why it matters, and what an abnormal value looks like.

36.1 Why Monitor?

┌─────────────────────────────────────────────────────────────────┐
│                    MONITORING OBJECTIVES                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   1. PERFORMANCE                                                │
│      - Track sync progress                                      │
│      - Monitor resource usage                                   │
│                                                                  │
│   2. AVAILABILITY                                               │
│      - Detect node downtime                                     │
│      - Alert on failures                                        │
│                                                                  │
│   3. SECURITY                                                   │
│      - Detect unusual patterns                                  │
│      - Monitor access attempts                                   │
│                                                                  │
│   4. CAPACITY PLANNING                                          │
│      - Track storage growth                                      │
│      - Plan upgrades                                             │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Golden rule: a metric you don’t understand is a metric you can’t alert on. For every metric you scrape, you should be able to answer What is it? Why does it matter? What is abnormal? — that is exactly the format used below.

36.2 Enabling Metrics

Geth Metrics

geth --metrics --metrics.addr 0.0.0.0 --metrics.port 6060

Metrics Endpoints

http://localhost:6060/debug/metrics               # human-readable
http://localhost:6060/debug/metrics/prometheus    # Prometheus text format — use this

⚠️ The Prometheus scrape path is /debug/metrics/prometheus, not /metrics. Configure it in prometheus.yml with metrics_path (below). A wrong path silently scrapes nothing — a classic blackhole.

Other Clients

Client	Flag	Prometheus path
Geth	`--metrics --metrics.addr`	`/debug/metrics/prometheus`
Reth	`--metrics <addr:port>`	`/metrics`
Erigon	`--metrics --metrics.addr`	`/debug/metrics/prometheus`
Besu	`--metrics-enabled --metrics-host --metrics-port`	`/metrics`
Nethermind	`http://<host>:<port>/metrics`	`/metrics`
Lighthouse (CL)	`--metrics` (`MetricsPort` default 5054)	`/metrics`
Prysm (CL)	`--monitoring-port` (default 8080)	`/metrics`
CometBFT/gaiad	`[instrumentation] prometheus = true`	`/metrics` (default 26660)
Solana	no first-party exporter — use RPC-derived signals (ch50)	n/a

36.3 Key Metrics to Track (WHAT / WHY / ABNORMAL)

Sync Metrics (Geth)

Metric	WHAT	WHY	ABNORMAL
`chain_head_block`	Number of the highest fully-imported block	The single most important “is my node alive” signal	Flat over 10+ min while peers exist → sync stalled
`chain_head_header`	Highest header seen on the network	Tells you how far the node is from the tip	Divergence from tip (compare with a public RPC)
`chain_head_reorg` (counter)	Blocks reverted due to chain reorgs	Frequent reorgs = peer/connectivity problems or bad peers	`increase(...[10m]) > 3` sustained → investigate

Transaction Pool (Geth)

Metric	WHAT	WHY	ABNORMAL
`txpool_pending`	Txs ready to be included in a block	Shows load and health of the mempool	Stuck at 0 while network is busy → not seeing txs (sync/peer issue)
`txpool_queued`	Txs waiting (nonce gaps, low fees)	Queue spikes indicate client-side batching or wallet misbehavior	Queued » pending relentlessly → nonce gaps, fee issues

P2P / Network (Geth)

Metric	WHAT	WHY	ABNORMAL
`p2p_peers`	Currently connected peers	Healthy node needs a stable peer set	Drops to < 5 → firewall, NAT, or failing peer ranking
`p2p_dials` (counter)	Peer connection attempts (in+out)	High dial rate = flapping connections	Sustained dial storms → peer_table churn, bad bootnodes
`p2p_ingress_traffic` / `p2p_egress_traffic` (counters)	Bytes received/sent	Verifies symmetric P2P flow; egress-heavy = serving lots of state	Ingres >> egress consistently → sync bottleneck or attack

RPC (Geth, when `--http` enabled)

Metric	WHAT	WHY	ABNORMAL
`rpc_duration` (histogram, label `method`)	Latency of JSON-RPC calls	Per-method p99 tells you which API is slow/abused	`eth_call`/`eth_getLogs` p99 breaching tolerance → cap or cache them
`rpc_failure` (counter)	Failed RPC invocations	Error-rate spikes indicate client bugs, timeouts, or abuse	Sudden 10x increase → check client + proxy logs

Runtime / Host (Geth Go runtime + node_exporter)

Metric	WHAT	WHY	ABNORMAL
`process_cpu_seconds_total`	Cumulative CPU seconds	Sustained high CPU during snapsync is normal; high CPU after sync is suspect	Pegged at 100% hours after sync → compaction or API abuse
`process_resident_memory_bytes`	RSS memory	Watch for leaks / cache bloat	Creeping past your budget with no cache change → investigate
`go_goroutines`	Live goroutines	Sudden spike often precedes OOM	> 5,000 and rising → goroutine leak
`node_filesystem_avail_bytes` (node_exporter)	Free disk	Disk fills kill nodes; pruning is cheaper than resyncing	Under 10% of disk → alert now (ch38)
`node_network_` / `node_disk_` (node_exporter)	Host NIC/disk stats	Detect NIC saturation and IOPS exhaustion	Disk `%util` near 100% while CPU idle → compaction storm

⚠️ Do not use eth_syncing as a Prometheus metric. It is a JSON-RPC method, not a metric. For “sync stalled” alerts use increase(chain_head_block[10m]) == 0 (ch38). Metric names above are geth names — Reth, Erigon, Nethermind, Besu and CL clients export different names; verify with curl localhost:6060/debug/metrics/prometheus | grep before writing rules.

36.4 Prometheus Configuration

Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
./prometheus --config.file=prometheus.yml --web.listen-address=:9090

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'geth'
    metrics_path: /debug/metrics/prometheus   # ← required, not /metrics
    static_configs:
      - targets: ['localhost:6060']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']            # node_exporter for host metrics

36.5 Grafana Dashboards

Import Dashboard

Install Grafana
Add Prometheus data source
Import dashboard:
- Ethereum (geth): community dashboards — check that the metric names match your client’s export before trusting the JSON
- Generic: create custom panels from the tables in 36.3

Key Dashboards

Dashboard	Description
Block Production	`chain_head_block` rate, reorg counter
Node Health	CPU, memory, disk (node_exporter)
Network	`p2p_peers`, `p2p_ingress/egress_traffic`, txpool
Consensus (CL)	Validator attestation/proposal metrics (Lighthouse `beacon_`, Prysm `validator_`)

36.6 Alerting

Prometheus Alerts (correct metric names)

groups:
  - name: geth-alerts
    rules:
      - alert: NodeDown
        expr: up{job="geth"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Geth node is down"

      - alert: NoPeers
        expr: p2p_peers == 0
        for: 5m
        labels:
          severity: warning

      - alert: SyncStalled
        expr: increase(chain_head_block[10m]) == 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Chain head not advancing"

      - alert: MempoolBacklog
        expr: txpool_queued > txpool_pending * 3
        for: 10m
        labels:
          severity: info

Full alert design (severity, routing, runbooks) is covered in Chapter 38: Alerting & Notifications.

Summary

Enable metrics with --metrics; scrape geth from /debug/metrics/prometheus (other clients differ!)
Every metric: know what it is, why it matters, what abnormal looks like
chain_head_block = the “is my node alive” signal; increase(...[10m]) == 0 = stalled sync
eth_syncing is an RPC method, not a metric — never use it in PromQL
Use Prometheus + Grafana; verify community dashboards against your client’s real metric names

Next Chapter

In Chapter 37: Logging & Log Management, we’ll explore logging.

Last Updated: 2026-09-28