AWS CloudTrail
Chapter 37: AWS CloudTrail - API Auditing
Section titled “Chapter 37: AWS CloudTrail - API Auditing”API Call Logging and Auditing
Section titled “API Call Logging and Auditing”37.1 Overview
Section titled “37.1 Overview”AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account.
AWS CloudTrail Overview+------------------------------------------------------------------+| || +------------------------+ || | AWS CloudTrail | || +------------------------+ || | || +---------------------+---------------------+ || | | | | || v v v v || +----------+ +----------+ +----------+ +----------+ || | Management| | Data | | Insights | | Lake | || | Events | | Events | | | | | || | | | | | | | | || | - Control | | - S3 | | - Anomaly| | - Query | || | Plane | | - Lambda | | - Detect | | - Analyze| || | - API | | - DynamoDB| | - Alert | | - Store | || +----------+ +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Key Features
Section titled “Key Features”| Feature | Description |
|---|---|
| Management Events | Control plane operations |
| Data Events | Resource-level operations |
| Insights | Anomaly detection |
| CloudTrail Lake | Query and analyze events |
37.2 Trail Configuration
Section titled “37.2 Trail Configuration”Trail Types
Section titled “Trail Types” CloudTrail Trail Types+------------------------------------------------------------------+| || Single-Region Trail || +------------------------------------------------------------+ || | | || | - Logs events from one region | || | - Lower cost | || | - Suitable for single-region workloads | || | | || +------------------------------------------------------------+ || || Multi-Region Trail || +------------------------------------------------------------+ || | | || | - Logs events from all regions | || | - Recommended for security auditing | || | - Single S3 bucket for all regions | || | | || +------------------------------------------------------------+ || || Organization Trail || +------------------------------------------------------------+ || | | || | - Logs events from all accounts in organization | || | - Centralized logging | || | - Created in management account | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Trail Configuration
Section titled “Trail Configuration” CloudTrail Trail Configuration+------------------------------------------------------------------+| || Storage Options || +------------------------------------------------------------+ || | | || | S3 Bucket: | || | +------------------------------------------------------+ | || | | - Primary storage for log files | | || | | - Object lock for immutability | | || | | - Lifecycle policies for retention | | || | +------------------------------------------------------+ | || | | || | CloudWatch Logs: | || | +------------------------------------------------------+ | || | | - Real-time log analysis | | || | | - Metric filters and alarms | | || | +------------------------------------------------------+ | || | | || | CloudTrail Lake: | || | +------------------------------------------------------+ | || | | - SQL query capability | | || | | - 7-year retention | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Log File Validation || +------------------------------------------------------------+ || | | || | - Digest files for integrity verification | || | - Detect log modification or deletion | || | - Required for compliance (PCI, HIPAA) | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+37.3 Event Types
Section titled “37.3 Event Types”Management Events
Section titled “Management Events” CloudTrail Management Events+------------------------------------------------------------------+| || Control Plane Operations || +------------------------------------------------------------+ || | | || | IAM Operations: | || | +------------------------------------------------------+ | || | | - CreateUser, DeleteUser | | || | | - AttachRolePolicy, DetachRolePolicy | | || | | - CreateAccessKey, DeleteAccessKey | | || | +------------------------------------------------------+ | || | | || | EC2 Operations: | || | +------------------------------------------------------+ | || | | - RunInstances, TerminateInstances | | || | | - CreateSecurityGroup, AuthorizeSecurityGroupIngress | | || | | - CreateVolume, DeleteVolume | | || | +------------------------------------------------------+ | || | | || | S3 Operations: | || | +------------------------------------------------------+ | || | | - CreateBucket, DeleteBucket | | || | | - PutBucketPolicy, DeleteBucketPolicy | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Read vs Write Events || +------------------------------------------------------------+ || | | || | Read-only: | || | +------------------------------------------------------+ | || | | - Describe*, Get*, List* | | || | | - No resource modification | | || | +------------------------------------------------------+ | || | | || | Write: | || | +------------------------------------------------------+ | || | | - Create*, Delete*, Update*, Put* | | || | | - Resource modification | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Data Events
Section titled “Data Events” CloudTrail Data Events+------------------------------------------------------------------+| || Supported Resources || +------------------------------------------------------------+ || | | || | Amazon S3: | || | +------------------------------------------------------+ | || | | - GetObject, PutObject, DeleteObject | | || | | - Object-level operations | | || | | - Can log all buckets or specific buckets | | || | +------------------------------------------------------+ | || | | || | AWS Lambda: | || | +------------------------------------------------------+ | || | | - Invoke function | | || | | - Can log all functions or specific functions | | || | +------------------------------------------------------+ | || | | || | Amazon DynamoDB: | || | +------------------------------------------------------+ | || | | - GetItem, PutItem, DeleteItem | | || | | - Table-level operations | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Note: Data events are not logged by default || +------------------------------------------------------------+ || | - Must be explicitly enabled | || | - Additional cost applies | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+37.4 CloudTrail Insights
Section titled “37.4 CloudTrail Insights”Insights Overview
Section titled “Insights Overview” CloudTrail Insights+------------------------------------------------------------------+| || Anomaly Detection || +------------------------------------------------------------+ || | | || | What it detects: | || | +------------------------------------------------------+ | || | | - Unusual API call volume | | || | | - Unusual API error rates | | || | | - Suspicious activity patterns | | || | +------------------------------------------------------+ | || | | || | How it works: | || | +------------------------------------------------------+ | || | | 1. Establishes baseline of normal activity | | || | | 2. Continuously monitors API calls | | || | | 3. Detects deviations from baseline | | || | | 4. Creates insight events | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Insight Types || +------------------------------------------------------------+ || | | || | ApiCallRateInsight: | || | +------------------------------------------------------+ | || | | - Unusual spike in API call volume | | || | | - Example: 100x normal TerminateInstances calls | | || | +------------------------------------------------------+ | || | | || | ApiErrorRateInsight: | || | +------------------------------------------------------+ | || | | - Unusual spike in API errors | | || | | - Example: Auth failures on AssumeRole | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+37.5 Event Structure
Section titled “37.5 Event Structure”CloudTrail Event Record
Section titled “CloudTrail Event Record”{ "eventVersion": "1.08", "userIdentity": { "type": "IAMUser", "principalId": "AIDACKCEVSQ6C2EXAMPLE", "arn": "arn:aws:iam::123456789012:user/alice", "accountId": "123456789012", "accessKeyId": "AKIAIOSFODNN7EXAMPLE", "userName": "alice" }, "eventTime": "2024-01-15T12:00:00Z", "eventSource": "ec2.amazonaws.com", "eventName": "StartInstances", "awsRegion": "us-east-1", "sourceIPAddress": "192.0.2.1", "userAgent": "aws-cli/2.0.0", "requestParameters": { "instancesSet": { "items": [ { "instanceId": "i-1234567890abcdef0" } ] } }, "responseElements": { "instancesSet": { "items": [ { "instanceId": "i-1234567890abcdef0", "currentState": { "code": 0, "name": "pending" }, "previousState": { "code": 80, "name": "stopped" } } ] } }, "resources": [ { "ARN": "arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0", "type": "AWS::EC2::Instance" } ], "eventType": "AwsApiCall", "recipientAccountId": "123456789012"}37.6 CLI Commands
Section titled “37.6 CLI Commands”# Create trailaws cloudtrail create-trail \ --name my-trail \ --s3-bucket-name my-cloudtrail-logs \ --include-global-service-events \ --is-multi-region-trail
# Start loggingaws cloudtrail start-logging \ --name my-trail
# Stop loggingaws cloudtrail stop-logging \ --name my-trail
# Get trail statusaws cloudtrail get-trail-status \ --name my-trail
# List trailsaws cloudtrail describe-trails
# Get trailaws cloudtrail get-trail \ --name my-trail
# Put event selectors (enable data events)aws cloudtrail put-event-selectors \ --trail-name my-trail \ --event-selectors '[ { "ReadWriteType": "All", "IncludeManagementEvents": true, "DataResources": [ { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::my-bucket/"] } ] } ]'
# Look up eventsaws cloudtrail lookup-events \ --lookup-attributes AttributeKey=Username,AttributeValue=alice \ --start-time 2024-01-01T00:00:00Z \ --end-time 2024-01-31T23:59:59Z
# Get event selectorsaws cloudtrail get-event-selectors \ --trail-name my-trail
# Enable insightsaws cloudtrail put-insight-selectors \ --trail-name my-trail \ --insight-selectors '[{"InsightType":"ApiCallRateInsight"}]'
# Delete trailaws cloudtrail delete-trail \ --name my-trail37.7 Best Practices
Section titled “37.7 Best Practices”CloudTrail Best Practices
Section titled “CloudTrail Best Practices” CloudTrail Best Practices+------------------------------------------------------------------+| || 1. Enable multi-region trails || +------------------------------------------------------------+ || | - Capture all regional activity | || | - Single source of truth | || +------------------------------------------------------------+ || || 2. Enable log file validation || +------------------------------------------------------------+ || | - Detect tampering | || | - Required for compliance | || +------------------------------------------------------------+ || || 3. Use S3 object lock || +------------------------------------------------------------+ || | - Write-once, read-many (WORM) | || | - Prevent deletion | || +------------------------------------------------------------+ || || 4. Enable CloudTrail Insights || +------------------------------------------------------------+ || | - Detect anomalous activity | || | - Proactive security monitoring | || +------------------------------------------------------------+ || || 5. Integrate with CloudWatch Logs || +------------------------------------------------------------+ || | - Real-time analysis | || | - Metric filters and alarms | || +------------------------------------------------------------+ || || 6. Use organization trails || +------------------------------------------------------------+ || | - Centralize logging for all accounts | || | - Simplify compliance | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+37.8 Why This Matters in DevOps/SRE
Section titled “37.8 Why This Matters in DevOps/SRE”CloudTrail is essential for security auditing, compliance, and incident investigation. SREs rely on it to understand who did what and when in the AWS environment.
CloudTrail in DevOps/SRE+------------------------------------------------------------------+| || SRE Security & Compliance: || || 1. Audit Trail || +----------------------------------------------------------+ || | - Every API call is logged with user, time, source IP | || | - Required for SOC2, HIPAA, PCI compliance | || | - Evidence for security investigations | || +----------------------------------------------------------+ || || 2. Incident Response || +----------------------------------------------------------+ || | - Trace unauthorized changes to source | || | - Identify compromised credentials | || | - Reconstruct timeline of security events | || +----------------------------------------------------------+ || || 3. Operational Troubleshooting || +----------------------------------------------------------+ || | - Understand who made configuration changes | || | - Correlate API activity with incidents | || | - Detect drift from expected configuration | || +----------------------------------------------------------+ || |+------------------------------------------------------------------+37.9 Linux Systems Perspective
Section titled “37.9 Linux Systems Perspective”CloudTrail Automation from Arch Linux
Section titled “CloudTrail Automation from Arch Linux”# Lookup recent eventsaws cloudtrail lookup-events \ --lookup-attributes attributeKey=EventSource,attributeValue=ec2.amazonaws.com \ --max-results 10
# Query CloudTrail Lake with SQL#!/bin/bash# ~/bin/cloudtrail-audit.shset -euo pipefail
QUERY_STRING="SELECT eventTime, eventName, userIdentity.userName, sourceIPAddressFROM abc123f4-5678-90ab-cdef-EXAMPLEWHERE eventName = 'ConsoleLogin'ORDER BY eventTime DESC LIMIT 20"
QUERY_ID=$(aws cloudtrail start-query \ --query-string "$QUERY_STRING" \ --query 'QueryId' \ --output text)
aws cloudtrail get-query-results \ --query-id "$QUERY_ID"37.10 Common Mistakes & Anti-Patterns
Section titled “37.10 Common Mistakes & Anti-Patterns” CloudTrail Anti-Patterns+------------------------------------------------------------------+| || ❌ Mistake 1: Not Enabling All Regions || +----------------------------------------------------------+ || | Problem: Missing activity from non-primary regions | || | Impact: Incomplete audit trail, compliance gaps | || | Fix: Enable multi-region trail | || +----------------------------------------------------------+ || || ❌ Mistake 2: Not Encrypting Log Files || +----------------------------------------------------------+ || | Problem: Logs can be read by unauthorized users | || | Impact: Sensitive data exposure | || | Fix: Use SSE-KMS encryption for S3 bucket | || +----------------------------------------------------------+ || || ❌ Mistake 3: Not Enabling Log File Validation || +----------------------------------------------------------+ || | Problem: Can't detect log tampering | || | Impact: Can't prove logs haven't been modified | || | Fix: Enable digest log validation | || +----------------------------------------------------------+ || || ❌ Mistake 4: Not Using Organization Trails || +----------------------------------------------------------+ || | Problem: Missing activity from member accounts | || | Impact: Incomplete security visibility | || | Fix: Create organization trail in management account | || +----------------------------------------------------------+ || |+------------------------------------------------------------------+37.11 Interview Questions
Section titled “37.11 Interview Questions”Conceptual Questions
Section titled “Conceptual Questions”-
Q: What’s the difference between management events and data events in CloudTrail?
- A: Management events (also called control plane) include operations like creating VPCs, starting instances, modifying security groups. Data events (data plane) include operations on data in services like S3 object access, Lambda executions, DynamoDB item operations. Data events are disabled by default due to volume.
-
Q: How does CloudTrail Insights work?
- A: CloudTrail Insights analyzes normal API activity patterns and detects anomalies. It identifies unusual activity like spikes in API calls, resource provisioning at unusual times, or operations from unusual locations. It generates Insights events when anomalies are detected, stored separately from regular events.
Scenario-Based Questions
Section titled “Scenario-Based Questions”- Q: You notice unauthorized resource creation in your account. How would you investigate with CloudTrail?
- A: Use CloudTrail: (1) Lookup events by resource type and creation time, (2) Identify the IAM principal that made the changes via userIdentity, (3) Check for the source IP to determine if internal or external, (4) Review subsequent events to see what else was modified, (5) Correlate with CloudWatch logs for the timeframe.
37.12 Exam Tips
Section titled “37.12 Exam Tips”
Key Exam Points+------------------------------------------------------------------+| || 1. CloudTrail logs API calls (management and data events) || || 2. Management events are logged by default || || 3. Data events must be explicitly enabled || || 4. Multi-region trails capture all regional activity || || 5. Organization trails capture all account activity || || 6. Log file validation ensures integrity || || 7. Insights detect anomalous API activity || || 8. CloudTrail Lake enables SQL queries on events || || 9. Events are delivered within 15 minutes || || 10. S3 with object lock prevents log tampering || |+------------------------------------------------------------------+37.13 Summary
Section titled “37.13 Summary” Chapter 37 Summary+------------------------------------------------------------------+| || CloudTrail Core Concepts || +------------------------------------------------------------+ || | - Trails: Configuration for logging | || | - Events: API call records | || | - Insights: Anomaly detection | || +------------------------------------------------------------+ || || Event Types || +------------------------------------------------------------+ || | - Management events: Control plane operations | || | - Data events: Resource-level operations | || | - Insights events: Anomalous activity | || +------------------------------------------------------------+ || || Best Practices || +------------------------------------------------------------+ || | - Enable multi-region trails | || | - Enable log file validation | || | - Use S3 object lock | || | - Enable Insights | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Previous Chapter: Chapter 36: Amazon CloudWatch - Metrics & Alarms Next Chapter: Chapter 38: AWS X-Ray - Distributed Tracing