AWS CodePipeline & CodeBuild
Chapter 32: AWS CodePipeline & CI/CD Best Practices
Section titled “Chapter 32: AWS CodePipeline & CI/CD Best Practices”Continuous Delivery Orchestration
Section titled “Continuous Delivery Orchestration”32.1 Overview
Section titled “32.1 Overview”AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
AWS CodePipeline Overview+------------------------------------------------------------------+| || +------------------------+ || | AWS CodePipeline | || +------------------------+ || | || +---------------------+---------------------+ || | | | || v v v || +----------+ +----------+ +----------+ || | Stages | | Actions | | Transitions || | | | | | | || | - Source | | - Source | | - Manual | || | - Build | | - Build | | - Auto | || | - Deploy | | - Deploy | | - Conditions || +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Pipeline Structure
Section titled “Pipeline Structure” CodePipeline Structure+------------------------------------------------------------------+| || Pipeline || +------------------------------------------------------------+ || | | || | Stage 1: Source Stage 2: Build Stage 3: Deploy || | +----------------+ +----------------+ +----------------+ || | | Action 1 | --> | Action 1 | --> | Action 1 | || | | (CodeCommit) | | (CodeBuild) | | (CodeDeploy) | || | +----------------+ +----------------+ +----------------+ || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+32.2 Pipeline Components
Section titled “32.2 Pipeline Components”Stages
Section titled “Stages” CodePipeline Stages+------------------------------------------------------------------+| || Source Stage || +------------------------------------------------------------+ || | | || | Supported Sources: | || | +------------------------------------------------------+ | || | | - AWS CodeCommit | | || | | - GitHub | | || | | - Bitbucket | | || | | - Amazon S3 | | || | | - Amazon ECR | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Build Stage || +------------------------------------------------------------+ || | | || | Supported Build Providers: | || | +------------------------------------------------------+ | || | | - AWS CodeBuild | | || | | - Jenkins | | || | | - TeamCity | | || | | - CloudBees | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || || Deploy Stage || +------------------------------------------------------------+ || | | || | Supported Deploy Providers: | || | +------------------------------------------------------+ | || | | - AWS CodeDeploy (EC2, Lambda, ECS) | | || | | - AWS CloudFormation | | || | | - AWS Elastic Beanstalk | | || | | - AWS ECS | | || | | - AWS S3 | | || | | - AWS AppConfig | | || | +------------------------------------------------------+ | || | | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Action Types
Section titled “Action Types” CodePipeline Action Types+------------------------------------------------------------------+| || Source Actions || +------------------------------------------------------------+ || | - Checkout code from repository | || | - Trigger pipeline on changes | || | - Output: source artifacts | || +------------------------------------------------------------+ || || Build Actions || +------------------------------------------------------------+ || | - Compile source code | || | - Run tests | || | - Output: build artifacts | || +------------------------------------------------------------+ || || Deploy Actions || +------------------------------------------------------------+ || | - Deploy to target environment | || | - Update infrastructure | || | - Output: deployment status | || +------------------------------------------------------------+ || || Approval Actions || +------------------------------------------------------------+ || | - Manual approval gate | || | - Require human review | || | - Output: approval status | || +------------------------------------------------------------+ || || Test Actions || +------------------------------------------------------------+ || | - Run automated tests | || | - Integration tests | || | - Output: test results | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+32.3 Pipeline Configuration
Section titled “32.3 Pipeline Configuration”Pipeline Definition (JSON)
Section titled “Pipeline Definition (JSON)” CodePipeline Definition+------------------------------------------------------------------+| || { || "pipeline": { || "name": "MyAppPipeline", || "roleArn": "arn:aws:iam::123456789012:role/CodePipelineRole", || "artifactStore": { || "type": "S3", || "location": "my-pipeline-artifacts" || }, || "stages": [ || { || "name": "Source", || "actions": [ || { || "name": "SourceAction", || "actionTypeId": { || "category": "Source", || "owner": "AWS", || "provider": "CodeCommit", || "version": "1" || }, || "outputArtifacts": [ || { "name": "SourceOutput" } || ], || "configuration": { || "RepositoryName": "my-repo", || "BranchName": "main" || } || } || ] || }, || { || "name": "Build", || "actions": [ || { || "name": "BuildAction", || "actionTypeId": { || "category": "Build", || "owner": "AWS", || "provider": "CodeBuild", || "version": "1" || }, || "inputArtifacts": [ || { "name": "SourceOutput" } || ], || "outputArtifacts": [ || { "name": "BuildOutput" } || ], || "configuration": { || "ProjectName": "my-build-project" || } || } || ] || }, || { || "name": "Deploy", || "actions": [ || { || "name": "DeployAction", || "actionTypeId": { || "category": "Deploy", || "owner": "AWS", || "provider": "CodeDeploy", || "version": "1" || }, || "inputArtifacts": [ || { "name": "BuildOutput" } || ], || "configuration": { || "ApplicationName": "my-app", || "DeploymentGroupName": "my-deployment-group" || } || } || ] || } || ] || } || } || |+------------------------------------------------------------------+32.4 Advanced Pipeline Patterns
Section titled “32.4 Advanced Pipeline Patterns”Multi-Stage Pipeline
Section titled “Multi-Stage Pipeline” Multi-Stage Pipeline Pattern+------------------------------------------------------------------+| || +----------+ +----------+ +----------+ +----------+ || | Source | -->| Build | -->| Test | -->| Approval | || | | | | | | | | || | CodeCommit| | CodeBuild| | CodeBuild| | Manual | || +----------+ +----------+ +----------+ +----------+ || | || v || +----------+ +----------+ +----------+ || | Prod | <--| Staging | <--| Deploy | || | Deploy | | Deploy | | Dev | || | | | | | | || | CodeDeploy| | CodeDeploy| | CodeDeploy| || +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Blue/Green Deployment Pipeline
Section titled “Blue/Green Deployment Pipeline” Blue/Green Deployment Pipeline+------------------------------------------------------------------+| || +----------+ +----------+ +----------+ +----------+ || | Source | -->| Build | -->| Test | -->| Create | || | | | | | | | Green | || +----------+ +----------+ +----------+ +----------+ || | || v || +----------+ +----------+ +----------+ || | Terminate| <--| Shift | <--| Validate | || | Blue | | Traffic | | Green | || +----------+ +----------+ +----------+ || |+------------------------------------------------------------------+Cross-Account Pipeline
Section titled “Cross-Account Pipeline” Cross-Account Pipeline+------------------------------------------------------------------+| || Dev Account Prod Account || +------------------+ +------------------+ || | | | | || | +----------+ | | +----------+ | || | | Source | | | | Deploy | | || | | (CodeCommit) | | | (CodeDeploy) | || | +----------+ | | +----------+ | || | | | | ^ | || | v | | | | || | +----------+ | | | | || | | Build | | | | | || | | (CodeBuild) | | | | || | +----------+ | | | | || | | | | | | || | +---------+------------>+--------+ | || | | | | || +------------------+ +------------------+ || |+------------------------------------------------------------------+32.5 CodePipeline CLI Commands
Section titled “32.5 CodePipeline CLI Commands”# Create pipelineaws codepipeline create-pipeline \ --cli-input-json file://pipeline-definition.json
# Get pipelineaws codepipeline get-pipeline \ --name my-pipeline
# List pipelinesaws codepipeline list-pipelines
# Update pipelineaws codepipeline update-pipeline \ --cli-input-json file://pipeline-definition.json
# Start pipeline executionaws codepipeline start-pipeline-execution \ --name my-pipeline
# List pipeline executionsaws codepipeline list-pipeline-executions \ --pipeline-name my-pipeline
# Get pipeline executionaws codepipeline get-pipeline-execution \ --pipeline-name my-pipeline \ --pipeline-execution-id execution-id
# Retry stage executionaws codepipeline retry-stage-execution \ --pipeline-name my-pipeline \ --stage-name Build \ --pipeline-execution-id execution-id \ --retry-mode FAILED_ACTIONS
# Stop pipeline executionaws codepipeline stop-pipeline-execution \ --pipeline-name my-pipeline \ --pipeline-execution-id execution-id
# Put approval resultaws codepipeline put-approval-result \ --pipeline-name my-pipeline \ --stage-name Approval \ --action-name ApprovalAction \ --result summary="Approved",status=Approved \ --token approval-token
# Delete pipelineaws codepipeline delete-pipeline \ --name my-pipeline32.6 CI/CD Best Practices
Section titled “32.6 CI/CD Best Practices”Pipeline Design Best Practices
Section titled “Pipeline Design Best Practices” Pipeline Design Best Practices+------------------------------------------------------------------+| || 1. Keep pipelines simple and focused || +------------------------------------------------------------+ || | - Single responsibility per pipeline | || | - Separate pipelines for different environments | || +------------------------------------------------------------+ || || 2. Implement proper artifact management || +------------------------------------------------------------+ || | - Use versioned artifacts | || | - Store artifacts in S3 with lifecycle policies | || +------------------------------------------------------------+ || || 3. Use manual approvals for production || +------------------------------------------------------------+ || | - Require human approval before production deployment | || | - Include relevant stakeholders | || +------------------------------------------------------------+ || || 4. Implement automated testing || +------------------------------------------------------------+ || | - Unit tests in build stage | || | - Integration tests in test stage | || | - Security scanning | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Security Best Practices
Section titled “Security Best Practices” CI/CD Security Best Practices+------------------------------------------------------------------+| || 1. Use least privilege IAM roles || +------------------------------------------------------------+ || | - Separate roles for each service | || | - Use resource-level permissions | || +------------------------------------------------------------+ || || 2. Secure secrets management || +------------------------------------------------------------+ || | - Use Secrets Manager or Parameter Store | || | - Never store secrets in code | || | - Rotate secrets regularly | || +------------------------------------------------------------+ || || 3. Enable encryption || +------------------------------------------------------------+ || | - Encrypt artifacts at rest | || | - Use KMS for encryption keys | || +------------------------------------------------------------+ || || 4. Implement audit logging || +------------------------------------------------------------+ || | - Enable CloudTrail for all CI/CD services | || | - Monitor pipeline execution logs | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Performance Best Practices
Section titled “Performance Best Practices” CI/CD Performance Best Practices+------------------------------------------------------------------+| || 1. Optimize build time || +------------------------------------------------------------+ || | - Use caching for dependencies | || | - Parallelize tests | || | - Use appropriate compute types | || +------------------------------------------------------------+ || || 2. Minimize artifact size || +------------------------------------------------------------+ || | - Only include necessary files | || | - Use .gitignore and exclude patterns | || +------------------------------------------------------------+ || || 3. Use incremental deployments || +------------------------------------------------------------+ || | - Deploy only changed components | || | - Use rolling updates | || +------------------------------------------------------------+ || || 4. Implement proper monitoring || +------------------------------------------------------------+ || | - Set up CloudWatch alarms | || | - Monitor pipeline execution duration | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+32.7 Integration with Other Services
Section titled “32.7 Integration with Other Services”EventBridge Integration
Section titled “EventBridge Integration” EventBridge Integration+------------------------------------------------------------------+| || CodeCommit Event --> EventBridge --> CodePipeline || || Event Pattern: || +------------------------------------------------------------+ || | { | || | "source": ["aws.codecommit"], | || | "detail-type": ["CodeCommit Repository State Change"], | || | "detail": { | || | "event": ["push"], | || | "repositoryName": ["my-repo"], | || | "referenceName": ["main"] | || | } | || | } | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+SNS Notifications
Section titled “SNS Notifications” SNS Notification Integration+------------------------------------------------------------------+| || Pipeline Events --> SNS Topic --> Subscribers || || Notification Types: || +------------------------------------------------------------+ || | - Pipeline execution started | || | - Pipeline execution succeeded | || | - Pipeline execution failed | || | - Stage execution started/succeeded/failed | || | - Action execution started/succeeded/failed | || | - Manual approval needed | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+32.8 Troubleshooting
Section titled “32.8 Troubleshooting”Common Issues
Section titled “Common Issues” Common CodePipeline Issues+------------------------------------------------------------------+| || Issue 1: Pipeline stuck in progress || +------------------------------------------------------------+ || | Cause: Action timeout or resource unavailable | || | Solution: Check action logs, increase timeout | || +------------------------------------------------------------+ || || Issue 2: Artifact not found || +------------------------------------------------------------+ || | Cause: Incorrect artifact name or missing output | || | Solution: Verify artifact names match between stages | || +------------------------------------------------------------+ || || Issue 3: Permission denied || +------------------------------------------------------------+ || | Cause: Missing IAM permissions | || | Solution: Check service role permissions | || +------------------------------------------------------------+ || || Issue 4: Source change not triggering pipeline || +------------------------------------------------------------+ || | Cause: Webhook not configured or disabled | || | Solution: Check CloudWatch Events rule for source | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+32.9 Why This Matters in DevOps/SRE
Section titled “32.9 Why This Matters in DevOps/SRE”CodePipeline is the orchestration engine for AWS CI/CD. Understanding its patterns is critical for SREs managing deployment workflows and reliability.
CodePipeline in DevOps/SRE+------------------------------------------------------------------+| || SRE Pipeline Principles: || || 1. Deployment Safety & Reliability || +----------------------------------------------------------+ || | - Pipeline as code: version-controlled deployment | || | - Automated gates: tests, security scans, approvals | || | - Blue-green/canary for zero-downtime releases | || +----------------------------------------------------------+ || || 2. Observability Integration || +----------------------------------------------------------+ || | - Pipeline emits CloudWatch metrics on success/failure | || | - SNS notifications for on-call alerts | || | - X-Ray tracing for distributed deployment analysis | || +----------------------------------------------------------+ || || 3. Error Budget & Release Engineering || +----------------------------------------------------------+ || | - Pipeline respects error budget policies | || | - Auto-rollback on SLO violations | || | - Feature flags for gradual rollouts | || +----------------------------------------------------------+ || |+------------------------------------------------------------------+32.10 Linux Systems Perspective
Section titled “32.10 Linux Systems Perspective”CodePipeline Automation from Arch Linux
Section titled “CodePipeline Automation from Arch Linux”# Install AWS CLI and tools on Arch Linuxsudo pacman -S aws-cli-v2 jq docker gityay -S aws-copilot
# Pipeline monitoring script#!/bin/bash# ~/bin/pipeline-monitor.shset -euo pipefail
PIPELINE_NAME="${1:-my-pipeline}"
echo "=== Pipeline: $PIPELINE_NAME ==="aws codepipeline get-pipeline-state \ --name "$PIPELINE_NAME" \ --query 'stageStates[].{Stage:stageName,Status:latestExecution.status}' \ --output table
echo ""echo "=== Latest Execution ==="aws codepipeline list-pipeline-executions \ --pipeline-name "$PIPELINE_NAME" \ --max-results 1 \ --query 'pipelineExecutionSummaries[0].{Status:status,Start:startTime,LastUpdate:lastUpdateTime}' \ --output table
# Watch pipeline progresswatch -n 10 'aws codepipeline get-pipeline-state --name my-pipeline --query "stageStates[].{Stage:stageName,Status:latestExecution.status}" --output table'32.11 Common Mistakes & Anti-Patterns
Section titled “32.11 Common Mistakes & Anti-Patterns” CodePipeline Anti-Patterns+------------------------------------------------------------------+| || ❌ Mistake 1: No Cross-Region Replication || +----------------------------------------------------------+ || | Problem: Pipeline in one region, apps in another | || | Impact: Latency, data sovereignty issues, failover gaps | || | Fix: Deploy pipelines in each region or use cross-region| || +----------------------------------------------------------+ || || ❌ Mistake 2: Hardcoded Artifact S3 Buckets || +----------------------------------------------------------+ || | Problem: Bucket names embedded in pipeline config | || | Impact: Cross-account deployments fail | || | Fix: Use pipeline variables and cross-account IAM | || +----------------------------------------------------------+ || || ❌ Mistake 3: No Manual Approvals for Production || +----------------------------------------------------------+ || | Problem: Auto-deploy to production on every commit | || | Impact: Unreviewed changes reach production | || | Fix: Add approval actions before production stages | || +----------------------------------------------------------+ || || ❌ Mistake 4: Ignoring Pipeline Execution History || +----------------------------------------------------------+ || | Problem: No logging of who approved what and when | || | Impact: Audit failures, incident investigation delays | || | Fix: Enable CloudTrail for pipeline events | || +----------------------------------------------------------+ || |+------------------------------------------------------------------+32.12 Interview Questions
Section titled “32.12 Interview Questions”Conceptual Questions
Section titled “Conceptual Questions”-
Q: How does CodePipeline integrate with CodeBuild and CodeDeploy?
- A: CodePipeline orchestrates the workflow: source stage pulls code → CodeBuild compiles/tests → CodeDeploy deploys to compute services. Artifacts are stored in S3 between stages. Pipeline can trigger on source changes (CodeCommit, S3, GitHub) and invoke Lambda for custom actions.
-
Q: Explain the difference between sequential and parallel actions in CodePipeline.
- A: Sequential actions run one after another within a stage (e.g., build then test). Parallel actions run simultaneously (e.g., multiple test suites). Parallel actions reduce execution time but require careful dependency management. Transitions between stages can be conditional based on action results.
Scenario-Based Questions
Section titled “Scenario-Based Questions”- Q: Design a multi-region deployment pipeline.
- A: Use CodePipeline with cross-region actions: source in primary region → build in primary → deploy to staging → manual approval → parallel deploy to multiple regions using CodeDeploy. Each region has its own pipeline or uses StackSets for infrastructure. Enable pipeline CloudTrail for audit. Use Route53 weighted routing for traffic shifting.
32.13 Exam Tips
Section titled “32.13 Exam Tips”
Key Exam Points+------------------------------------------------------------------+| || 1. CodePipeline orchestrates the entire CI/CD workflow || || 2. Pipelines consist of stages, actions, and transitions || || 3. Each stage can have multiple actions (sequential/parallel) || || 4. Artifacts are passed between stages via S3 || || 5. Manual approval actions require human intervention || || 6. EventBridge triggers pipelines on source changes || || 7. Cross-account pipelines use KMS encryption || || 8. CodePipeline supports third-party integrations || || 9. Use retry-stage-execution to retry failed actions || || 10. Pipeline execution can be stopped mid-execution || |+------------------------------------------------------------------+32.14 Summary
Section titled “32.14 Summary” Chapter 32 Summary+------------------------------------------------------------------+| || AWS CodePipeline || +------------------------------------------------------------+ || | - Fully managed continuous delivery service | || | - Visual workflow for release process | || | - Integrates with AWS and third-party services | || +------------------------------------------------------------+ || || Pipeline Components || +------------------------------------------------------------+ || | - Stages: Source, Build, Test, Deploy, Approval | || | - Actions: Individual operations within stages | || | - Artifacts: Files passed between stages | || | - Transitions: Movement between stages | || +------------------------------------------------------------+ || || Best Practices || +------------------------------------------------------------+ || | - Implement automated testing | || | - Use manual approvals for production | || | - Secure secrets properly | || | - Monitor pipeline performance | || +------------------------------------------------------------+ || |+------------------------------------------------------------------+Previous Chapter: Chapter 31: AWS CodeCommit, CodeBuild & CodeDeploy Next Chapter: Chapter 33: AWS CloudFormation - Infrastructure as Code