SSH Hardening, Key Management & Bastion Hosts

Chapter 20: SSH Hardening, Key Management & Bastion Hosts

Learning Objectives

Harden SSH to production security standards
Implement proper SSH key management and certificate infrastructure
Design and operate bastion hosts / jump servers
Understand SSH tunneling and port forwarding

Prerequisites

Chapter 15: System Hardening

What is this? (Beginner On-Ramp)

SSH (Secure Shell) is how administrators remotely control Linux servers. SSH hardening is the practice of securing this critical doorway by disabling passwords, requiring cryptographic keys, changing default settings, and blocking brute-force attacks.

20.1 SSH Security Model

    SSH Authentication Flow
    ────────────────────────

    Client                              Server (sshd)
    ──────                              ─────────────
    ssh user@server
         │
         │ TCP connect to port 22
         ▼
    Key Exchange (ECDH/DH):
    Client ──── client_key_share ────►  Server
    Client ◄─── server_key_share ────   Server
    Both derive: session_key (never transmitted!)
         │
         │ All further communication encrypted
         ▼
    Server Authentication:
    Server ──── server_host_key_sig ──► Client
    Client verifies: Is this the real server?
    (checks ~/.ssh/known_hosts)
         │
         ▼
    User Authentication (one of):
    • Public key: Client signs challenge with private key
    • Password: encrypted password
    • Keyboard-interactive: OTP, etc.
         │
         ▼
    Shell/Command session starts

20.2 sshd_config: Complete Hardening

# /etc/ssh/sshd_config — Production Hardened Configuration

# ─── Protocol and Port ────────────────────────────────────
Port 22                          # Consider changing (security through obscurity)
Protocol 2                       # SSH Protocol 2 only (Protocol 1 is broken)
AddressFamily inet               # IPv4 only (or any for IPv6 too)
ListenAddress 0.0.0.0           # Listen on all interfaces (or specific IP)

# ─── Cryptography (FIPS/CIS Compliant) ───────────────────
# Key Exchange Algorithms (prefer ECDH, use strong DH)
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,diffie-hellman-group-exchange-sha256

# Host Key Algorithms (prefer Ed25519, ECDSA)
HostKeyAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521

# Encryption (ChaCha20 and AES-GCM preferred)
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

# MACs (Encrypt-then-MAC preferred)
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

# ─── Authentication ───────────────────────────────────────
LoginGraceTime 60                # Disconnect if no auth in 60s
MaxAuthTries 3                   # 3 tries before disconnect
MaxSessions 10                   # Max concurrent sessions
PermitRootLogin no               # NEVER allow root login directly
PermitEmptyPasswords no          # Disable empty passwords

# Key-based auth
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# Password auth (disable in production when possible)
PasswordAuthentication no        # Require key-based auth
ChallengeResponseAuthentication no
UsePAM yes                       # Use PAM for account/session handling

# Keyboard-interactive (needed for 2FA/TOTP)
KbdInteractiveAuthentication no  # Set to yes only if using 2FA

# ─── Access Control ───────────────────────────────────────
# Restrict which users/groups can SSH in
AllowUsers deploy alice bob      # Explicit allowlist
# OR:
AllowGroups ssh-users            # Group-based allowlist
DenyUsers root admin             # Explicit denylist

# ─── Features ─────────────────────────────────────────────
# Forwarding options (disable what you don't need)
AllowTcpForwarding no            # Disable TCP tunneling
AllowAgentForwarding no          # Disable SSH agent forwarding
AllowStreamLocalForwarding no    # Disable Unix socket forwarding
GatewayPorts no                  # No remote port forwarding
X11Forwarding no                 # No X11 forwarding
PermitTunnel no                  # No VPN tunneling

# For bastion hosts, enable forwarding selectively:
# AllowTcpForwarding local        # Allow local forwarding only
# Match Group ssh-jump-users
#     AllowTcpForwarding yes
#     PermitOpen *:22             # Allow only SSH to destination

# ─── Session Settings ─────────────────────────────────────
ClientAliveInterval 300          # Send keepalive every 5 minutes
ClientAliveCountMax 3            # Disconnect after 3 missed keepalives
# (15 minutes idle = disconnect)

TCPKeepAlive yes                 # TCP keepalive (different from SSH keepalive)
Compression delayed              # Compress after auth (no pre-auth compression)

# ─── Logging ──────────────────────────────────────────────
SyslogFacility AUTH
LogLevel VERBOSE                 # Log keys used, not just user/host

# ─── SFTP Subsystem (if needed, chroot for security) ─────
Subsystem sftp internal-sftp     # Use built-in SFTP
# Match Group sftp-users
#     ChrootDirectory /data/sftp/%u
#     ForceCommand internal-sftp
#     AllowTcpForwarding no

# ─── Host Keys ────────────────────────────────────────────
HostKey /etc/ssh/ssh_host_ed25519_key     # Primary (preferred)
HostKey /etc/ssh/ssh_host_ecdsa_key       # Fallback
# Remove: HostKey /etc/ssh/ssh_host_rsa_key  (only if RSA not needed)
# Remove: HostKey /etc/ssh/ssh_host_dsa_key  (DSA is broken)

# Test the configuration
sshd -t                          # Test syntax
sshd -T                          # Display full parsed config

# Check supported algorithms
ssh -Q kex                       # Supported key exchange
ssh -Q cipher                    # Supported ciphers
ssh -Q mac                       # Supported MACs
ssh -Q key                       # Supported key types

# Apply changes
systemctl reload sshd

20.3 SSH Key Management

Generating Strong Keys

# ── Generate Keys ─────────────────────────────────────────
# Ed25519 (recommended — compact, fast, secure)
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -C "user@hostname-$(date +%Y)" -N "passphrase"

# ECDSA (if Ed25519 not supported)
ssh-keygen -t ecdsa -b 521 -f ~/.ssh/id_ecdsa -C "user@hostname"

# RSA (legacy systems that don't support Ed25519)
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa -C "user@hostname"

# NEVER use: DSA (1024-bit, broken), RSA-1024, RSA-2048 without rotation

# ── Verify Key ────────────────────────────────────────────
ssh-keygen -l -f ~/.ssh/id_ed25519.pub
# 256 SHA256:xxxxx user@hostname-2024 (ED25519)
# Shows: bits, fingerprint, comment, type

# ── Deploy Public Key ─────────────────────────────────────
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server
# Or manually:
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

authorized_keys Options

# Options can restrict key usage:

# Restrict to specific source IP
from="10.0.1.0/24,192.168.1.5" ssh-ed25519 AAAA...

# Allow only specific command (deploy key)
command="/usr/local/bin/deploy.sh",no-port-forwarding,no-pty,no-x11-forwarding ssh-ed25519 AAAA...

# Restrict to port forwarding only (for tunnel users)
no-pty,no-user-rc,no-x11-forwarding,permitopen="db.internal:5432" ssh-ed25519 AAAA...

# Certificate with expiry (using SSH CA — see below)
cert-authority,principals="alice" ssh-ed25519 AAAA... CA_key_comment

20.4 SSH Certificate Authority

SSH CAs solve the scaling problem of managing authorized_keys files across hundreds of servers.

    Traditional Key Management Problem
    ───────────────────────────────────

    100 servers × 50 users = 5,000 authorized_keys entries to manage
    Each new user requires 100 server updates
    Each fired employee requires 100 server updates (immediately!)

    SSH CA Solution:
    ───────────────
    Servers trust the CA public key (one entry per server)
    Users get signed certificates from the CA

    User wants access:
    1. User uploads their public key to CA (once)
    2. CA signs it with CA private key (creates certificate)
    3. User SSH's to server with their private key + certificate
    4. Server verifies certificate signature against CA public key
    5. Access granted (based on certificate validity + principals)

    Revocation:
    → Add revoked key/cert fingerprint to KRL (Key Revocation List)
    → All servers check KRL before allowing access
    → Instant revocation across the entire fleet!

# ── Setup CA ──────────────────────────────────────────────
# Generate CA key pair (KEEP THIS OFFLINE/IN HSM!)
ssh-keygen -t ed25519 -f /etc/ssh/ca/ssh_ca -C "SSH CA 2024"

# ── Configure Servers to Trust CA ─────────────────────────
# /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/ca/ssh_ca.pub

# ── Sign User Keys ────────────────────────────────────────
# Sign a user's public key
ssh-keygen -s /etc/ssh/ca/ssh_ca \
    -I "alice-2024-01-15" \           # Certificate identity
    -n "alice,developer,web-prod" \   # Allowed principals (usernames)
    -V +30d \                          # Valid for 30 days
    -z 42 \                            # Serial number
    ~/.ssh/alice_id_ed25519.pub

# Creates: alice_id_ed25519-cert.pub

# View certificate details
ssh-keygen -L -f alice_id_ed25519-cert.pub
# Type: ssh-ed25519-cert-v01@openssh.com user certificate
# Public key: ED25519-CERT SHA256:xxx
# Signing CA: ED25519 SHA256:yyy (using rsa-sha2-512)
# Key ID: "alice-2024-01-15"
# Serial: 42
# Valid: from 2024-01-15T00:00:00 to 2024-02-14T00:00:00
# Principals: alice, developer, web-prod
# Critical Options: (none)
# Extensions: permit-pty, permit-user-rc, permit-agent-forwarding

# ── User SSH with Certificate ──────────────────────────────
# SSH automatically uses certificate if same basename as key
ssh -i ~/.ssh/alice_id_ed25519 alice@server   # Uses alice_id_ed25519-cert.pub

# ── Revoke Certificates ───────────────────────────────────
# Create a KRL (Key Revocation List)
ssh-keygen -k -f /etc/ssh/ca/revoked_keys -z 1 alice_id_ed25519.pub

# sshd_config:
RevokedKeys /etc/ssh/ca/revoked_keys

# Distribute to all servers (via Ansible/puppet)
ansible all -m copy -a "src=/etc/ssh/ca/revoked_keys dest=/etc/ssh/ca/revoked_keys"

20.5 Bastion Hosts / Jump Servers

    Bastion Host Architecture
    ──────────────────────────

    Internet / Developer Workstations
          │
          │ SSH to bastion (port 22 open)
          ▼
    ┌────────────────────────────────────────────────┐
    │            BASTION HOST (DMZ)                   │
    │                                                  │
    │  • Hardened, minimal OS                         │
    │  • Only SSH daemon running                      │
    │  • All logins logged/audited                    │
    │  • 2FA required                                 │
    │  • No outbound internet                         │
    │  • Auto-rotate host keys                        │
    └───────────────────────┬────────────────────────┘
                            │ SSH jump to internal
                            ▼
    ┌────────────────────────────────────────────────┐
    │            PRIVATE NETWORK                      │
    │                                                  │
    │  web-01  web-02  db-01  redis-01  k8s-nodes    │
    │  (no direct SSH from internet)                  │
    └────────────────────────────────────────────────┘

# ── Using Bastion Host (Jump Host) ────────────────────────
# Method 1: -J flag (modern, preferred)
ssh -J bastion.example.com user@internal-server-01

# Method 2: ProxyJump in ~/.ssh/config
cat >> ~/.ssh/config << 'EOF'
Host bastion
    Hostname bastion.example.com
    User ec2-user
    IdentityFile ~/.ssh/bastion_key

Host *.internal
    User ubuntu
    IdentityFile ~/.ssh/internal_key
    ProxyJump bastion

Host db.internal
    User postgres
    ProxyJump bastion
    LocalForward 5432 localhost:5432    # Forward DB port locally
EOF

# Now connect transparently:
ssh web-01.internal         # Goes through bastion automatically
ssh db.internal             # Port 5432 available locally after connect

Bastion Host Hardening

# ── Bastion-specific sshd_config additions ────────────────
# Force all sessions through audit wrapper
ForceCommand /usr/local/bin/audit_ssh_wrapper.sh

# Restrict users to jump only (no shell)
# Match group jumpers:
#     AllowTcpForwarding yes
#     PermitOpen *:22
#     ForceCommand echo "Jump connections only"

# Log all bastion sessions with script
# audit_ssh_wrapper.sh:
#!/bin/bash
SESSION_LOG="/var/log/ssh-sessions/session-$(whoami)-$(date +%Y%m%d-%H%M%S).log"
script -q -f -c "${SSH_ORIGINAL_COMMAND:-bash}" "$SESSION_LOG"

20.6 Two-Factor Authentication with SSH

# Install Google Authenticator PAM
apt-get install libpam-google-authenticator

# Configure for each user
google-authenticator
# Generates TOTP secret, scan QR code with authenticator app

# /etc/pam.d/sshd — Add 2FA
auth required pam_google_authenticator.so

# /etc/ssh/sshd_config
KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive  # Require BOTH key AND TOTP

# Result: User must have valid SSH key AND valid TOTP code
systemctl restart sshd

20.7 Interview Questions

Q1: Why is PermitRootLogin no important and how do you access root when needed?

Answer: PermitRootLogin no prevents direct SSH logins as root, reducing the attack surface. If an attacker guesses or steals root credentials, they still can’t SSH in directly. For legitimate admin tasks: (1) SSH as your personal user account (never share accounts — this breaks audit trails). (2) Use sudo on the target server to perform privileged actions. (3) If root shell is absolutely needed, sudo -i or sudo su -. This model ensures every action is attributable to a specific person via sudo logs and auid in the audit log.

Q2: What is an SSH certificate authority and what problems does it solve?

Answer: An SSH CA is a key pair where the private key signs user (or host) public keys, creating certificates. It solves the fleet management problem: instead of maintaining an authorized_keys file on every server listing every user’s key, each server only needs the CA’s public key as TrustedUserCAKeys. When a user’s certificate (signed by the CA) is presented, the server trusts it. Benefits: (1) Instant revocation — add to KRL, distribute, all servers reject immediately. (2) Time-bounded access — certificates have expiry dates; users must renew. (3) Auditable — certificates have a signed identity and serial number. (4) Onboarding scale — new server just gets the CA public key, no per-server user management.

20.8 Summary

Security Feature	Configuration
Disable root login	`PermitRootLogin no`
Key-only auth	`PasswordAuthentication no`
Restrict users	`AllowUsers` or `AllowGroups`
Disable forwarding	`AllowTcpForwarding no`
Session timeout	`ClientAliveInterval 300`
Strong ciphers	Modern KexAlgorithms, Ciphers, MACs
Certificate auth	`TrustedUserCAKeys`
Bastion host	`ProxyJump` in SSH config
2FA	Google Authenticator PAM

Next Chapter: Chapter 21: Secure Boot, TPM & Disk Encryption

Prerequisites

Basic networking and SSH knowledge.

Advantages & Disadvantages

Advantages: Secures the primary administrative doorway, prevents brute-force botnets. Disadvantages: Losing SSH keys locks you out, strict IP whitelisting breaks remote work.

Common Mistakes

Locking yourself out by disabling password auth before setting up keys.
Leaving SSH port open to the internet without rate limiting (fail2ban) or IP whitelisting.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
SSH connection refused or timeout	Firewall blocking or sshd not running	`systemctl status sshd; iptables -L`	Allow port 22 in firewall; start sshd
Permission denied (publickey)	Wrong key, bad permissions on ~/.ssh	`ssh -v user@host`	chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys

Hands-on Lab

Objective: Harden an SSH server.

# 1. Edit sshd_config
# sudo vi /etc/ssh/sshd_config
# Ensure:
# PermitRootLogin no
# PasswordAuthentication no
# AllowUsers myuser

# 2. Test configuration before restarting
sshd -t

# 3. Reload sshd
systemctl reload sshd

Exercises

Configure SSH to run on a non-standard port and require public key authentication.
Set up a Bastion host using ProxyJump in your local ~/.ssh/config to access an internal server.

Revision Notes

Never permit root login over SSH.
Use Ed25519 keys instead of RSA if possible.
Bastion hosts limit exposure of internal network resources.
Use fail2ban to protect against brute-force attacks.

SSH Hardening, Key Management & Bastion Hosts

Chapter 20: SSH Hardening, Key Management & Bastion Hosts

Learning Objectives

Prerequisites

What is this? (Beginner On-Ramp)

20.1 SSH Security Model

20.2 sshd_config: Complete Hardening

20.3 SSH Key Management

Generating Strong Keys

authorized_keys Options

20.4 SSH Certificate Authority

20.5 Bastion Hosts / Jump Servers

Bastion Host Hardening

20.6 Two-Factor Authentication with SSH

20.7 Interview Questions

20.8 Summary

Next Chapter: Chapter 21: Secure Boot, TPM & Disk Encryption

Prerequisites

Advantages & Disadvantages

Common Mistakes

Troubleshooting

Hands-on Lab

Exercises

Revision Notes

Further Reading

Related Chapters