Skip to content
Flowchart Learning

Intrusion Detection and Fail2Ban

Chapter 34: Intrusion Detection and Fail2Ban

Section titled “Chapter 34: Intrusion Detection and Fail2Ban”

Comprehensive Guide to Linux Intrusion Detection

Section titled “Comprehensive Guide to Linux Intrusion Detection”

Why This Matters in DevOps/SRE

Section titled “Why This Matters in DevOps/SRE”

Intrusion detection is essential for proactive security:

                 IDS for DevOps/SRE
+------------------------------------------------------------------+
|                                                                   |
|    Real-time Protection:                                            |
|    +----------------------------------------------------------+  |
|    | Fail2Ban → Blocks brute force in real-time             |  |
|    | OSSEC → Monitors file integrity, logs                  |  |
|    | RKHunter → Detects rootkits                            |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Compliance & Forensics:                                          |
|    +----------------------------------------------------------+  |
|    | AIDE → File integrity for compliance                   |  |
|    | Lynis → Security auditing                              |  |
|    | Audit logs → Evidence for investigations              |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Incident Response:                                               |
|    +----------------------------------------------------------+  |
|    | IDS alerts → Early detection of compromises           |  |
|    | File integrity → Detect post-breach changes           |  |
|    | Log analysis → Understand attack vectors              |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+

Practical Impact:

  • Detect and block attacks automatically
  • Meet compliance requirements (PCI-DSS requires IDS)
  • Investigate security incidents with evidence
  • Detect file tampering (malware, backdoors)

34.1 Understanding Intrusion Detection

Section titled “34.1 Understanding Intrusion Detection”

Types of IDS/IPS

Section titled “Types of IDS/IPS”
                 Intrusion Detection Systems
+------------------------------------------------------------------+
|                                                                   |
|    Network-based IDS (NIDS):                                       |
|    +----------------------------------------------------------+   |
|    | • Monitors network traffic                                |   |
|    | • Examples: Snort, Suricata                             |   |
|    | • Passive monitoring                                    |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Host-based IDS (HIDS):                                         |
|    +----------------------------------------------------------+   |
|    | • Monitors system activity                               |   |
|    | • Examples: OSSEC, AIDE                                 |   |
|    | • File integrity, process monitoring                    |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Intrusion Prevention System (IPS):                              |
|    +----------------------------------------------------------+   |
|    | • Active blocking of attacks                            |   |
|    | • Examples: Fail2Ban, Snort-inline                     |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Detection Methods:                                              |
|    +----------------------------------------------------------+   |
|    | Signature-based | Known attack patterns                   |   |
|    | Anomaly-based   | Deviations from normal                |   |
|    | Stateful        | Protocol state analysis               |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

34.2 Fail2Ban Configuration

Section titled “34.2 Fail2Ban Configuration”

Installation and Setup

Section titled “Installation and Setup”
Terminal window
# Install
sudo apt install fail2ban          # Debian/Ubuntu
sudo yum install fail2ban         # RHEL/CentOS
sudo pacman -S fail2ban           # Arch


# Start service
sudo systemctl enable --now fail2ban


# Configuration file
# /etc/fail2ban/jail.local (recommended)
# /etc/fail2ban/jail.conf (default)

jail.local Configuration

Section titled “jail.local Configuration”
/etc/fail2ban/jail.local
[DEFAULT]
# Ban time (duration)
bantime = 1h


# Time window for retries
findtime = 10m


# Max retries before ban
maxretry = 3


# Email notifications
destemail = admin@example.com
sender = fail2ban@example.com
action = %(action_mwl)s


# Whitelist (never ban)
ignoreip = 127.0.0.1/8 10.0.0.0/8 192.168.0.0/16

SSH Jail

Section titled “SSH Jail”
/etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 1h
findtime = 10m
action = %(action_mwl)s


# Custom port
[sshd-custom]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3

Web Server Jails

Section titled “Web Server Jails”
Terminal window
# Nginx HTTP Auth
[nginx-http-auth]
enabled = true
port = http,https
filter = nginx-http-auth
logpath = /var/log/nginx/error.log
maxretry = 5


# Nginx Bot Search
[nginx-badrequests]
enabled = true
port = http,https
filter = nginx-badrequests
logpath = /var/log/nginx/error.log


# Apache Auth
[apache-auth]
enabled = true
port = http,https
filter = apache-auth
logpath = /var/log/apache2/error.log
maxretry = 5


# Apache Badbots
[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
logpath = /var/log/apache2/error.log

Mail Server Jails

Section titled “Mail Server Jails”
Terminal window
# Postfix
[postfix]
enabled = true
port = smtp,submission,imap3,imaps,pop3,pop3s
filter = postfix
logpath = /var/log/mail.log


# Dovecot
[dovecot]
enabled = true
port = smtp,submission,imap3,imaps,pop3,pop3s
filter = dovecot
logpath = /var/log/mail.log


# SASL
[sasl]
enabled = true
port = smtp,submission,imap3,imaps,pop3,pop3s
filter = sasl
logpath = /var/log/mail.log
maxretry = 3

Custom Jail Example

Section titled “Custom Jail Example”
Terminal window
# Block repeated login attempts to API
[api-rate-limit]
enabled = true
port = http,https
filter = api-rate-limit
logpath = /var/log/api/access.log
maxretry = 10
bantime = 30m
findtime = 1m
action = %(action_mwl)s


# Create filter
# /etc/fail2ban/filter.d/api-rate-limit.conf
[Definition]
failregex = ^<HOST> .* "POST /api/login
ignoreregex =

Fail2Ban Commands

Section titled “Fail2Ban Commands”
Terminal window
# Status
sudo fail2ban-client status
sudo fail2ban-client status sshd


# Ban/Unban
sudo fail2ban-client set sshd banip 1.2.3.4
sudo fail2ban-client set sshd unbanip 1.2.3.4


# Reload
sudo fail2ban-client reload
sudo fail2ban-client reload sshd


# Check logs
tail -f /var/log/fail2ban.log


# Test filter
fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

34.3 OSSEC (Host-based IDS)

Section titled “34.3 OSSEC (Host-based IDS)”

Installation

Section titled “Installation”
Terminal window
# Install
sudo apt install ossec-hids           # Debian/Ubuntu
sudo yum install ossec-hids           # RHEL/CentOS
sudo pacman -S ossec-hids            # Arch


# Configure
sudo /var/ossec/bin/ossec-configure

Configuration

Section titled “Configuration”
/var/ossec/etc/ossec.conf
# Local rules
# /var/ossec/rules/local_rules.xml


# Commands
# View alerts
tail -f /var/ossec/logs/alerts/alerts.log


# Manage
/var/ossec/bin/ossec-control start
/var/ossec/bin/ossec-control stop
/var/ossec/bin/ossec-control restart


# Client sync
/var/ossec/bin/manage_agents

34.4 AIDE (File Integrity)

Section titled “34.4 AIDE (File Integrity)”

Installation

Section titled “Installation”
Terminal window
# Install
sudo apt install aide           # Debian/Ubuntu
sudo yum install aide          # RHEL/CentOS
sudo pacman -S aide           # Arch

Initialization and Usage

Section titled “Initialization and Usage”
Terminal window
# Initialize database
sudo aide --init


# Rename database
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db


# Check integrity
sudo aide --check


# Update database
sudo aide --update


# Check specific file
sudo aide --check /etc/passwd


# Daily check (cron)
sudo crontab -e
0 5 * * * /usr/bin/aide --check

Configuration

Section titled “Configuration”
/etc/aide/aide.conf
# Database location
database=file:/var/lib/aide/aide.db
database_out=file:/var/lib/aide/aide.db.new


# Groups
/etc/pam.d R
/etc/passwd R
/etc/shadow R
/etc/group R
/etc/sudoers R
/var/log R
/bin R
/sbin R
/usr/bin R
/usr/sbin R
/etc/cron R
/etc/hosts R

34.5 RKHunter (Rootkit Detection)

Section titled “34.5 RKHunter (Rootkit Detection)”

Installation and Usage

Section titled “Installation and Usage”
Terminal window
# Install
sudo apt install rkhunter          # Debian/Ubuntu
sudo yum install rkhunter          # RHEL/CentOS


# Update
sudo rkhunter --update


# Check system
sudo rkhunter --check
sudo rkhunter --check --sk


# Properties test
sudo rkhunter --propupd

34.6 Lynis (Security Auditing)

Section titled “34.6 Lynis (Security Auditing)”

Installation and Usage

Section titled “Installation and Usage”
Terminal window
# Install
sudo apt install lynis            # Debian/Ubuntu
sudo yum install lynis           # RHEL/CentOS
sudo pacman -S lynis            # Arch


# Run audit
sudo lynis audit system
sudo lynis audit system --cronjob


# Check hardening
sudo lynis audit system --details


# Quick scan
sudo lynis quick

34.7 Interview Questions

Section titled “34.7 Interview Questions”

Basic Questions

Section titled “Basic Questions”

  1. What is Fail2Ban?

    • Intrusion prevention system that bans IPs after failed attempts

  2. What is the difference between IDS and IPS?

    • IDS detects; IPS prevents (active blocking)

  3. What is AIDE?

    • File integrity monitoring tool

  4. How does Fail2Ban work?

    • Monitors log files, regex matching, bans via iptables

  5. What is OSSEC?

    • Host-based intrusion detection system

Intermediate Questions

Section titled “Intermediate Questions”

  1. What are the main components of Fail2Ban?

    • Filters, Actions, Jails

  2. How do you whitelist an IP in Fail2Ban?

    • Add to ignoreip in jail.local

  3. What is the purpose of findtime in Fail2Ban?

    • Time window for counting retries

  4. How do you create a custom Fail2Ban jail?

    • Create filter in /etc/fail2ban/filter.d/ and jail in jail.local

  5. What is rootkit detection?

    • Detecting hidden malicious software

Common Mistakes & Anti-Patterns

Section titled “Common Mistakes & Anti-Patterns”

1. Not Monitoring IDS Alerts

Section titled “1. Not Monitoring IDS Alerts”
Terminal window
# ❌ WRONG: Installing IDS but ignoring alerts
# Fail2Ban bans but never checks who's being banned
# No review of OSSEC alerts


# ✅ CORRECT: Set up alerting
# Fail2Ban: Configure email notifications
action = %(action_mwl)s


# OSSEC: Email alerts
<email_notification>yes</email_notification>
<email_to>security@example.com</email_to>

2. Not Maintaining AIDE Database

Section titled “2. Not Maintaining AIDE Database”
Terminal window
# ❌ WRONG: AIDE database never updated
# After legitimate changes, everything looks like an alert!


# ✅ CORRECT: Update database after changes
aide --update  # After system updates
aide --check    # Regular checks
# Schedule in cron
0 3 * * * /usr/sbin/aide --check

3. Overly Permissive Fail2Ban Jails

Section titled “3. Overly Permissive Fail2Ban Jails”
Terminal window
# ❌ WRONG: Too aggressive banning
bantime = -1  # Permanent ban!
findtime = 1  # 1 minute!
maxretry = 1  # Single mistake = ban


# ✅ CORRECT: Balanced settings
bantime = 3600   # 1 hour
findtime = 600   # 10 minutes
maxretry = 5     # 5 attempts

4. Not Running Regular Scans

Section titled “4. Not Running Regular Scans”
Terminal window
# ❌ WRONG: Only running RKHunter when breached
# Should run regularly as proactive measure


# ✅ CORRECT: Schedule regular scans
# /etc/cron.d/rkhunter
0 3 * * * /usr/bin/rkhunter --check --cronjob


# Lynis weekly
exec 0 4 * * 0 /usr/bin/lynis audit system --cron

Summary

Section titled “Summary”
                 Quick Reference
+------------------------------------------------------------------+
|                                                                   |
|    Fail2Ban:                                                      |
|    +----------------------------------------------------------+  |
|    | sudo fail2ban-client status           | Status             |  |
|    | sudo fail2ban-client set sshd banip  | Ban IP             |  |
|    | sudo fail2ban-client set sshd unbanip | Unban IP           |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    AIDE:                                                          |
|    +----------------------------------------------------------+  |
|    | sudo aide --init                  | Initialize            |  |
|    | sudo aide --check                 | Check integrity      |  |
|    | sudo aide --update                | Update DB            |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    OSSEC:                                                         |
|    +----------------------------------------------------------+  |
|    | /var/ossec/bin/ossec-control start| Start                |  |
|    | tail -f /var/ossec/logs/alerts/ | View alerts          |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+