Alerting

Chapter 38: Alerting & Notifications

Overview

Metrics tell you what is happening, logs tell you why, but alerting is what tells you now — when something needs a human. This chapter covers how to design effective alert rules, configure Alertmanager, route notifications, and avoid the two classic failures of alerting: alert fatigue (too many noisy alerts) and blackholing (no alert when it matters).

---

38.1 The Monitoring Stack

Alerting sits at the top of a monitoring hierarchy:

┌─────────────────────────────────────────────────────────────────┐
│                    MONITORING PYRAMID                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│                    ┌─────────────────┐                           │
│                    │    ALERTS       │   "Wake someone up"       │
│                    │  (Chapter 38)   │                           │
│                    └────────┬────────┘                           │
│                             ▼                                    │
│                    ┌─────────────────┐                           │
│                    │   DASHBOARDS    │   "What's happening now?" │
│                    │  (Chapter 39)   │                           │
│                    └────────┬────────┘                           │
│                             ▼                                    │
│              ┌─────────────────────────────┐                     │
│              │  METRICS + LOGS + HEALTH    │  "Raw signals"      │
│              │  (Chapters 36, 37, 40)      │                     │
│              └─────────────────────────────┘                     │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

The key difference: metrics and dashboards are passive (someone must look), alerts are active (they look for you). A node can be down for hours while dashboards are green — because nobody opened them. Alerting exists to remove the “nobody looked” failure mode.

---

38.2 Alert Design Principles

The 4 Properties of a Good Alert

Property	Meaning	Example
Actionable	A human can do something about it	”Node out of disk space” → add disk
Timely	Fires before the failure hurts users	Alert at 85% disk, not 99.9%
Accurate	Few false positives	Confirm with 2 consecutive samples
Proportional	Severity matches impact	Page for slashing risk, email for low peers

Severity Levels

Severity	Meaning	Response	Example
P0 / Critical	Node down, funds at risk, validator missing duties	Page immediately (24/7)	Validator missed 5+ consecutive attestations
P1 / High	Service degraded, will fail soon	Page within 15 min	Disk 85% full, sync lag growing
P2 / Medium	Degraded but stable	Email/Slack, next business day	Peer count low but stable
P3 / Low	Informational	Ticket / dashboard only	Sync took longer than average

Rules of Thumb

• Alert on symptoms, not causes. Alert on “sync stalled” (symptom), not on “CPU at 90%” (cause — CPU at 90% during snap sync is fine).
• Predict, don’t report. Alert when disk will fill, not when it is full. Prometheus predict_linear exists for this.
• One runbook per alert. Every alert rule should have a link to a runbook (Chapter 45-46 troubleshooting is your runbook source).
• Keep the noise low. If an alert fires and nobody acts on it, delete or downgrade it. Every noisy alert trains people to ignore alerts.

---

38.3 Prometheus Alert Rules

Structure of the Rules File

rules.yml is loaded via the Prometheus --config.file / rule_files directive:

# prometheus.yml — add at the top level:
rule_files:
  - "/etc/prometheus/rules/*.yml"

/etc/prometheus/rules/nodes.yml

groups:
  - name: ethereum-node
    rules:
      # Node fell behind or stalled — the single most important check
      - alert: NodeSyncStalled
        expr: increase(chain_head_block[10m]) == 0
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} has not imported a block in 15m"
          description: "Chain head is stuck at {{ $value }}. Check peers and logs."
          runbook: "docs/troubleshooting.md#sync-stalled"

Real Geth Metric Names

Alert Scenario	PromQL Expression (geth --metrics)	Notes
Sync stalled	increase(chain_head_block[10m]) == 0	Head block not advancing
Sync falling behind	(time() - chain_head_block_timestamp) > 60	Requires a block-timestamp textfile/exported metric
No peers	p2p_peers < 5	Geth exposes p2p_peers
Memory pressure	(process_resident_memory_bytes / 32e9) > 0.85	Compare against node RAM budget
High goroutines	go_goroutines > 5000	Sudden growth often precedes OOM
RPC too slow	histogram_quantile(0.99, rate(rpc_duration_bucket[5m])) > 2	geth rpc_duration histogram (labels: method)
Reorg storm	increase(chain_head_reorg[10m]) > 3	Frequent reorgs → connectivity/peer issues

⚠️ Use your client’s real metric names. Geth exports chain_head_block, p2p_peers, txpool_pending, rpc_duration — but Reth, Erigon, Nethermind and Besu have different names. Always verify with curl http://localhost:6060/debug/metrics/prometheus | grep before writing rules. Fabricated names produce rules that never fire (silent blackholes!).

Sync-Lag Alert (Advanced)

To alert on “node is behind the network tip” (not just stalled), scrape the tip from a trusted external source and expose it via a textfile collector:

# every minute, cron job:
curl -s -X POST http://localhost:8545 \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  | jq -r '.result' | sed 's/^0x//' | awk '{print strtonum("0x"$1)}' \
  > /var/lib/node-exporter/node_tip.prom  # needs --collector.textfile.directory

rules.yml

- alert: NodeBehindTip
  expr: chain_head_block < textfile_node_tip - 20
  for: 5m
  labels: { severity: high }

---

38.4 Alertmanager

Alertmanager receives firing alerts from Prometheus and handles grouping, silencing, and routing to notification channels.

                  ┌──────────────────────────────┐
   Prometheus ───▶│         ALERTMANAGER         │
                  │  - Group related alerts      │
                  │  - Apply silences            │
                  │  - Route by severity/labels  │
                  └──────────┬───────────────────┘
                             │
                ┌────────────┼────────────┐
                ▼            ▼            ▼
           ┌─────────┐  ┌─────────┐  ┌─────────┐
           │ Slack   │  │ Email   │  │ Webhook │
           │         │  │         │  │ (Opsgenie/PagerDuty)
           └─────────┘  └─────────┘  └─────────┘

Installation

# Docker
docker run -d \
  --name alertmanager \
  -p 9093:9093 \
  -v /etc/alertmanager:/etc/alertmanager \
  prom/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml

Routing Configuration

/etc/alertmanager/alertmanager.yml

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s          # collect related alerts for 30s
  group_interval: 5m       # re-notify at most every 5m
  repeat_interval: 4h      # repeat pages at most every 4h
  receiver: 'slack-ops'
  routes:
    - matchers:
        - severity = critical
      receiver: 'page-24-7'
      continue: true
    - matchers:
        - alertname =~ ".*Validator.*"
      receiver: 'slack-validators'

receivers:
  - name: 'slack-ops'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T0000/B0000/XXXX'
        channel: '#node-alerts'
        send_resolved: true
  - name: 'page-24-7'
    webhook_configs:
      - url: 'https://events.pagerduty.com/integration/.../enqueue'
  - name: 'slack-validators'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T0000/B0000/XXXX'
        channel: '#validator-alerts'

Silencing Maintenance Windows

amtool is the CLI for Alertmanager. Use it during planned upgrades so maintenance doesn’t page people:

# Silence alert "NodeSyncStalled" for 2 hours during a hard-fork upgrade
amtool silence add \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h \
  --comment="Hard-fork upgrade window" \
  'alertname="NodeSyncStalled"'

---

38.5 Alerts by Node Type

Different node roles need different alert sets. A validator outage costs money; an RPC outage costs users.

Validator / Staking Node

Alert	Signal	Why it matters
Missing attestations	CL client metric (e.g., Lighthouse beacon_attestation_sent rate == 0 over 5m)	Directly reduces rewards; repeated misses can lead to offline penalties
Missed block proposal	beacon_block_proposal_sent no increase in your slot	Validator assigned to propose but failed
Being slashed / double-sign risk	Two beacon clients running, same keypair	Unrecoverable — mitigations from Chapter 35
EL-CL disconnected	EL client metric engine_* errors / CL execution-endpoint errors	Chain stalls, validator can’t build blocks

RPC / Public Node

Alert	Signal	Why it matters
High error rate	rate(http_requests_total{code=~"5.."}[5m]) > 0.01 (behind a proxy)	Users getting failures
Latency spike	histogram_quantile(0.99, rate(rpc_duration_bucket[5m])) > 2s	Slow queries time out
4xx abuse spike	rate(http_requests_total{code=~"4.."}[5m]) > N	Possible attack or client bug
Backend unhealthy	LB marks all backends down	Total outage — page immediately

Full Node (Private / Infrastructure)

Alert	Signal	Why it matters
Disk filling	node_filesystem_avail_bytes < 10% (node_exporter)	Full disk = corrupt/stopped node
No new blocks	increase(chain_head_block[10m]) == 0	Stalled or partitioned
Peer erosion	p2p_peers < 10 sustained	Network partition, firewall change
Process restarted	process_start_time_seconds changed (see Chapter 40)	Crash loops

---

38.6 Notification Channels Compared

Channel	Best for	Pros	Cons
Slack/Discord	Team awareness	Cheap, searchable history	Easy to ignore
Email	Medium severity	Simple, archived	Delayed, spam folder
Webhook → PagerDuty/Opsgenie	P0 critical	Escalation chains, acknowledge flows	Costly, must be reserved for real P0s
SMS/Phone	Last resort	Reliable	Expensive, ignored if overused

Best practice: route severity → channel (P0 → page, P1-P2 → Slack, P3 → dashboard). Never page for something a person can’t fix in the next hour.

---

38.7 Alert Response (Runbooks)

┌─────────────────────────────────────────────────────────────────┐
│                    ALERT RESPONSE FLOW                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ALERT FIRES ──▶ ACKNOWLEDGE (within 15 min)                    │
│                    │                                             │
│                    ▼                                             │
│              TRIAGE ──▶ Is it real?                              │
│                    │   Check dashboard + logs (ch36, 37)        │
│                    │                                             │
│                    ▼                                             │
│          ┌─────────┴─────────┐                                   │
│          │                   │                                   │
│      FIXABLE?            NOT FIXABLE?                            │
│          │                   │                                   │
│          ▼                   ▼                                   │
│   Apply runbook          Escalate to senior/                    │
│   (ch45-48)              vendor, monitor                        │
│          │                   │                                   │
│          ▼                   ▼                                   │
│   RESOLVE ──▶ Verify metric back to normal                     │
│   Close alert, note root cause                                  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

For every alert you should be able to answer in 60 seconds:

1. What is failing? (from the alert description)
2. Where? (instance label)
3. Since when? (firing timestamp)
4. Is it getting worse? (rate of the underlying metric)
5. What do we do first? (runbook link)

---

38.8 Common Pitfalls

Pitfall	Symptom	Fix
Alert fatigue	People silence every alert	Only keep alerts with a runbook; delete noisy ones
Blackhole rules	Rule never fires (wrong metric name)	Verify metric names against /debug/metrics/prometheus
Flapping	Alert toggles on/off	Add for: duration + hysteresis (alert at 0.85, clear at 0.80)
Threshold too late	Disk fills before alert fires	Alert on prediction: predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
No severity	Everything is “critical”	Nothing is critical; downgrade aggressively
Alerting on causes	CPU alerts waking people at 2am	Alert on symptoms (sync stalled, errors) instead

---

Summary

• Alerts are the only part of monitoring that actively seeks attention — design them like a product.
• Alert on symptoms (sync stalled, disk nearly full) with real metric names.
• Use Alertmanager to group, route by severity, and silence maintenance windows.
• Validator, RPC, and full nodes need different alert sets.
• Every alert needs a runbook and a 60-second answer to “what do we do now?”

---

Next Chapter

In Chapter 39: Grafana Dashboards, we’ll visualize these metrics and alerts on dashboards.

---

Last Updated: 2026-09-28