Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing AWS ECR Authentication Failures in GitHub Actions CI/CD Pipelines

As a Senior Cloud Solution Architect, I often encounter challenges related to secure and efficient CI/CD pipelines. One of the most common stumbling blocks for teams deploying containerized applications involves AWS ECR (Elastic Container Registry) authentication within GitHub Actions workflows. This comprehensive guide and troubleshooting manual will equip you with the knowledge to diagnose, resolve, and prevent these critical failures, ensuring your deployments run smoothly.

Symptom Analysis & Root Causes

Understanding the symptoms and underlying causes is the first step towards effective resolution. ECR authentication failures in GitHub Actions typically manifest as build or deployment errors when your workflow attempts to push or pull Docker images.

Common Symptoms and Error Messages:

• Error: Cannot perform an interactive login from a non TTY device: Often seen when docker login is attempted without proper token input.
• no basic auth credentials: Indicates that Docker tried to authenticate but found no valid username/password pair.
• Authentication required or denied: Your authorization token has expired. Reauthenticate and try again.: Clear indicators of invalid or expired AWS credentials/tokens.
• Error response from daemon: Get "https://[AWS_ACCOUNT_ID].dkr.ecr.[AWS_REGION].amazonaws.com/v2/": unauthorized: authentication required: Generic authentication failure from the Docker daemon.
• AWS credentials could not be found: The aws-actions/configure-aws-credentials action failed to acquire credentials.

Underlying Root Causes:

• Incorrect IAM Permissions: The AWS IAM Role assumed by GitHub Actions lacks the necessary permissions to call ecr:GetAuthorizationToken and other ECR actions (ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:PutImage, etc.).
• Misconfigured OpenID Connect (OIDC) Provider: The trust relationship between GitHub Actions and AWS IAM is not correctly established, preventing GitHub from assuming the specified IAM role. This could be due to an incorrect OIDC provider URL, Audience, or Subject.
• Expired ECR Authorization Token: ECR tokens are short-lived (12 hours). If credentials are hardcoded or improperly managed, the token may expire mid-workflow or between runs. OIDC mitigates this by generating fresh, temporary credentials for each run.
• Incorrect ECR Repository URI: Typo or incorrect region in the ECR repository URL when pushing or pulling images.
• GitHub Actions Workflow Configuration Errors: Misuse or incorrect parameters for actions like aws-actions/configure-aws-credentials and docker/login-action.
• Network Restrictions or Proxies: Less common in standard GitHub-hosted runners, but custom runners might have firewall or proxy rules blocking access to ECR endpoints.

Step-by-Step Resolution Guide

This guide assumes you are using AWS IAM Roles with OpenID Connect (OIDC) for authentication, which is the recommended and most secure approach for GitHub Actions.

Prerequisites:

• AWS Account with Administrator or IAM-capable permissions.
• GitHub Repository with admin access.
• AWS CLI and JQ installed (for local testing/verification).

Step 1: Verify GitHub Actions OIDC Configuration in AWS IAM

The OIDC provider allows GitHub to exchange its signed JWT tokens for temporary AWS credentials. Ensure it's correctly set up.

1. Check OIDC Provider Existence: Navigate to AWS IAM Console > Identity Providers. You should see a provider with the URL https://token.actions.githubusercontent.com.
2. Verify Thumbprint and Audience: The thumbprint should be current, and the audience should typically be sts.amazonaws.com.
3. If not present, create it:

# Create OIDC Provider (if it doesn't exist) # Replace [OIDC_THUMBPRINT] with the current GitHub OIDC thumbprint (e.g., d54058d25ad4cd55104b9015bc233e721d67a73d) # You can find the latest thumbprint at https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#adding-the-openid-connect-identity-provider-to-aws aws iam create-open-id-connect-provider \ --url https://token.actions.githubusercontent.com \ --thumbprint-list "[OIDC_THUMBPRINT]" \ --client-id-list sts.amazonaws.com

Also, ensure your GitHub Actions workflow includes the necessary permissions block:

# .github/workflows/your-workflow.yml name: Build and Push Docker Image on: push: branches: - main jobs: build-and-push: runs-on: ubuntu-latest permissions: id-token: write # This is crucial for OIDC contents: read steps: # ... rest of your steps

Step 2: Review AWS IAM Role and Permissions

The IAM Role that GitHub Actions assumes must have a Trust Policy configured for the OIDC provider and an attached policy granting ECR permissions.

1. Create or Update IAM Role: Go to AWS IAM Console > Roles. Create a new role or select an existing one dedicated to your GitHub Actions.
2. Configure Trust Policy: In the role, go to the "Trust relationships" tab and ensure it trusts the OIDC provider for your GitHub repository.

# Example Trust Policy for an IAM Role { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::[AWS_ACCOUNT_ID]:oidc-provider/token.actions.githubusercontent.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" }, "StringLike": { "token.actions.githubusercontent.com:sub": "repo:[YOUR_GITHUB_ORG]/[YOUR_REPO_NAME]:*" } } } ] }

Important: Adjust [YOUR_GITHUB_ORG]/[YOUR_REPO_NAME] to match your repository. For all repositories in an organization, you can use repo:[YOUR_GITHUB_ORG]/*:*.

3. Attach Permissions Policy: Attach an IAM policy to this role that grants the necessary ECR permissions.

# Example IAM Policy for ECR Access { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:GetLifecyclePolicy", "ecr:DescribeImages", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:GetRepositoryPolicy" ], "Resource": "arn:aws:ecr:[AWS_REGION]:[AWS_ACCOUNT_ID]:repository/[YOUR_ECR_REPO_NAME]" }, { "Effect": "Allow", "Action": [ "ecr:CompleteLayerUpload", "ecr:UploadLayerPart", "ecr:InitiateLayerUpload", "ecr:PutImage" ], "Resource": "arn:aws:ecr:[AWS_REGION]:[AWS_ACCOUNT_ID]:repository/[YOUR_ECR_REPO_NAME]" }, { "Effect": "Allow", "Action": "ecr:GetAuthorizationToken", "Resource": "*" # GetAuthorizationToken is a global action, usually needs "*" } ] }

Note: Replace [AWS_ACCOUNT_ID], [AWS_REGION], and [YOUR_ECR_REPO_NAME] with your specific values. Use least privilege; restrict resource ARNs where possible.

Step 3: Correct GitHub Actions Workflow Configuration

The GitHub Actions workflow itself needs to correctly assume the IAM role and then use the obtained credentials to log into ECR.

# .github/workflows/your-workflow.yml name: Build and Push to ECR on: push: branches: - main env: AWS_REGION: us-east-1 # Your AWS region ECR_REPOSITORY: your-app-repo # Your ECR repository name IMAGE_TAG: ${{ github.sha }} # Or any other tag jobs: build-and-push-image: runs-on: ubuntu-latest permissions: id-token: write # Required for OIDC contents: read steps: - name: Checkout repository uses: actions/checkout@v4 - name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::[AWS_ACCOUNT_ID]:role/[YOUR_GITHUB_ACTIONS_IAM_ROLE_NAME] aws-region: ${{ env.AWS_REGION }} - name: Login to Amazon ECR id: login-ecr uses: docker/login-action@v3 with: registry: ${{ fromJson(steps.configure-aws-credentials.outputs.aws-credentials).EcrRegistry }} # Dynamic registry lookup username: ${{ fromJson(steps.configure-aws-credentials.outputs.aws-credentials).EcrUsername }} password: ${{ fromJson(steps.configure-aws-credentials.outputs.aws-credentials).EcrPassword }} - name: Build and push Docker image uses: docker/build-push-action@v5 with: context: . push: true tags: | ${{ fromJson(steps.login-ecr.outputs.registry) }}/${{ env.ECR_REPOSITORY }}:${{ env.IMAGE_TAG }} ${{ fromJson(steps.login-ecr.outputs.registry) }}/${{ env.ECR_REPOSITORY }}:latest # Optional: push latest tag too

Key points in the workflow:

• permissions: id-token: write is critical for OIDC to work.
• aws-actions/configure-aws-credentials@v4: Assumes the IAM role. Make sure role-to-assume is the correct ARN.
• docker/login-action@v3: Uses the credentials provided by the AWS action to log into ECR. The dynamic lookup for registry, username, and password from configure-aws-credentials output is the most robust method.
• docker/build-push-action@v5: Builds and pushes your image. Ensure your tags are correctly formed, including the full ECR registry URI.

Step 4: Check for Expired Credentials/Tokens (Manual/Legacy Cases)

While OIDC largely eliminates this issue, if you're using a legacy approach (e.g., IAM user access keys stored as GitHub secrets), ensure the ECR authorization token is refreshed. The GetAuthorizationToken API call returns a token valid for 12 hours.

# Manual token refresh (for local debugging or legacy pipelines) aws ecr get-login-password --region [YOUR_REGION] | docker login --username AWS --password-stdin [AWS_ACCOUNT_ID].dkr.ecr.[YOUR_REGION].amazonaws.com

For GitHub Actions, always prefer OIDC, as it automatically handles token rotation.

Step 5: Network Connectivity and Proxy Issues (Self-Hosted Runners)

If you are using self-hosted GitHub Actions runners, verify that the runner can reach the AWS ECR endpoints (e.g., *.dkr.ecr.[region].amazonaws.com and api.ecr.[region].amazonaws.com). Check firewall rules, security groups, and proxy configurations if applicable.

Best Practices for Prevention & Performance Optimization

• Adopt OIDC: Always use IAM Roles with OpenID Connect for GitHub Actions. This eliminates the need to manage long-lived AWS access keys in GitHub Secrets, significantly enhancing security.
• Least Privilege Principle: Grant only the minimum necessary ECR permissions to your IAM role. Avoid using ecr:* or allowing access to all resources ("Resource": "*") unless absolutely required and justified.
• Specific Repository Access: In your IAM policy, specify the exact ECR repository ARN instead of allowing access to all repositories.
• Regular Audits: Periodically review your IAM policies, OIDC trust policies, and GitHub Actions workflows for outdated configurations or overly permissive access.
• Image Tagging Strategy: Implement a clear and consistent Docker image tagging strategy (e.g., using Git SHA, semantic versioning, or build numbers).
• CI/CD Caching: Use Docker layer caching in your build process to speed up subsequent builds and reduce ECR interactions when only minor changes occur.
• Error Logging: Ensure your GitHub Actions workflows have detailed logging enabled to capture all output, which is invaluable for debugging.

Frequently Asked Questions (FAQs)

Q1: Why is OIDC considered more secure than storing AWS access keys?

A: OIDC (OpenID Connect) provides short-lived, temporary credentials directly to the GitHub Actions runner by leveraging a trust relationship with AWS IAM. This eliminates the need to store static, long-lived AWS access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) as GitHub secrets, which are a common security vulnerability if compromised. With OIDC, credentials are valid only for the duration of the workflow run and are tied to specific GitHub repository and workflow contexts, enforcing a stronger identity perimeter.

Q2: My workflow fails with "AccessDenied" even after setting up OIDC and IAM roles. What could be wrong?

A: An "AccessDenied" error, especially after OIDC setup, typically points to issues with the IAM policy attached to the role. Double-check that:

1. The IAM policy specifically grants ecr:GetAuthorizationToken.
2. The policy includes all required ECR actions for pushing/pulling (e.g., ecr:PutImage, ecr:BatchCheckLayerAvailability) for the specific ECR repository ARN.
3. There are no implicit denies from other attached policies or SCPs (Service Control Policies) if you're in an AWS Organization.
4. The ECR repository itself doesn't have a repository policy that denies access to the assumed role.

Use AWS CloudTrail to examine the exact API calls being denied and the identity performing them, which can pinpoint the missing permission.

Q3: How do I handle multiple ECR repositories in different AWS regions or accounts from a single GitHub Actions workflow?

A: For multiple repositories, you can define different IAM roles for each, or craft a more generic IAM policy if repositories are within the same account and region. For different regions or accounts:

1. Different Regions (Same Account): Use separate aws-actions/configure-aws-credentials steps, each configuring credentials for a specific region. The OIDC role's policy would need to grant permissions across those regions.
2. Different Accounts: This requires a slightly more advanced setup. The GitHub Actions assumed role in Account A would need permission to sts:AssumeRole into a role in Account B. You would use one aws-actions/configure-aws-credentials step to assume the initial role in Account A, and then potentially another aws-actions/configure-aws-credentials step (or custom scripting) to assume the cross-account role in Account B, or manage credential environments carefully. Ensure trust policies and resource ARNs are correct for cross-account access.

Always ensure your IAM policies reflect the exact resources and actions required for each target ECR repository.