Fixing GitHub Actions Deployments to AWS EKS with OIDC Role Assumption Errors

Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing GitHub Actions Deployments to AWS EKS with OIDC Role Assumption Errors

Deploying applications to Amazon Elastic Kubernetes Service (EKS) using GitHub Actions is a powerful and efficient CI/CD strategy. Leveraging OpenID Connect (OIDC) for secure credential management eliminates the need for long-lived AWS access keys, significantly enhancing security. However, misconfigurations in OIDC trust policies, IAM roles, or GitHub Actions workflows can lead to frustrating AccessDenied or STS AssumeRole errors. This comprehensive guide, authored by a Senior Cloud Solution Architect and Software Engineer, provides a deep dive into diagnosing, troubleshooting, and resolving these common OIDC role assumption errors, ensuring your deployments to AWS EKS run smoothly.

Symptom Analysis & Root Causes

Understanding the symptoms and their underlying causes is the first step towards an effective resolution. When GitHub Actions fails to deploy to EKS using OIDC, you typically encounter specific error messages pointing to permission issues.

Common Error Messages

  • AccessDenied: User: arn:aws:sts::ACCOUNT_ID:assumed-role/github-actions-role/github-actions is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::ACCOUNT_ID:role/your-eks-deployment-role
    This indicates the IAM role configured in GitHub Actions is attempting to assume another role, but lacks the necessary permissions to do so. Often, this is a misconfiguration in the trust policy of the target IAM role.
  • An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
    This error means the GitHub Actions runner itself (via its OIDC token) is not allowed to perform the AssumeRoleWithWebIdentity action for the specified IAM role. This usually points to issues with the IAM role's trust policy.
  • WebIdentityTokenMismatch or InvalidIdentityToken
    These errors suggest issues with the OIDC token itself, often related to an incorrect OIDC provider URL, audience, or subject configuration in the IAM role's trust policy, or an improperly formed token.
  • ResourceNotFoundException: No OpenIDConnect provider found in your account for https://token.actions.githubusercontent.com
    This is a clear indicator that the OIDC provider for GitHub Actions has not been created or is incorrectly configured in your AWS account.
  • Error: Credentials could not be loaded, please check your action's inputs and/or AWS credentials configuration
    A generic error that can stem from any of the above, or a misconfiguration in the aws-actions/configure-aws-credentials action itself.

Underlying Causes

  • Missing or Incorrect AWS OIDC Provider: AWS needs to trust GitHub's OIDC issuer URL (https://token.actions.githubusercontent.com) to validate the identity tokens issued by GitHub Actions.
  • Incorrect IAM Role Trust Policy: The IAM role that GitHub Actions attempts to assume must have a trust policy that explicitly allows sts:AssumeRoleWithWebIdentity from the GitHub OIDC provider, with correct conditions (e.g., audience sts.amazonaws.com, subject matching the repository and workflow).
  • Insufficient Permissions on Assumed IAM Role: The IAM role assumed by GitHub Actions must have the necessary permissions to interact with EKS (e.g., eks:DescribeCluster, eks:UpdateKubeconfig, kubernetes:* via kubectl).
  • Misconfigured GitHub Actions Workflow:
    • Missing permissions: id-token: write.
    • Incorrect role-to-assume ARN in aws-actions/configure-aws-credentials.
    • Incorrect AWS region.
  • EKS aws-auth ConfigMap Issues: While the OIDC role assumption gets you AWS credentials, those credentials still need authorization within EKS. The aws-auth ConfigMap in your EKS cluster maps IAM roles/users to Kubernetes RBAC roles. If the assumed IAM role is not mapped, kubectl commands will fail with authorization errors.

Step-by-Step Resolution Guide

This guide walks you through verifying and correcting common misconfigurations. Ensure you have the AWS CLI configured with appropriate administrative permissions before proceeding.

Prerequisites

  • AWS CLI installed and configured.
  • kubectl installed.
  • Access to your AWS account with IAM administrator permissions.
  • Access to your GitHub repository with administrative permissions.
  • Your EKS cluster name and region.

Step 1: Verify EKS OIDC Provider

Your EKS cluster must have an associated OIDC identity provider to allow external OIDC providers (like GitHub Actions) to authenticate. First, check if your EKS cluster has an OIDC provider URL.

aws eks describe-cluster --name YOUR_EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text

This command should return an OIDC issuer URL for your EKS cluster (e.g., https://oidc.eks.REGION.amazonaws.com/id/EXAMPLED99EE4B6E8D7129532588D0123456789).
Next, ensure a GitHub Actions OIDC provider exists in IAM.

aws iam list-open-id-connect-providers | grep token.actions.githubusercontent.com

If no OIDC provider for GitHub is found, you need to create one. Replace <ACCOUNT_ID> with your AWS account ID.

# Get the thumbprint for GitHub's OIDC provider. It changes periodically. # For current thumbprint, check: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services # As of early 2024, common thumbprint is 6938fd4d98bab03faadb97b34396831e3780fae1 # Example if using OpenSSL: # openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout # Create the OIDC provider aws iam create-open-id-connect-provider \ --url https://token.actions.githubusercontent.com \ --thumbprint-list "6938fd4d98bab03faadb97b34396831e3780fae1" \ --client-id-list "sts.amazonaws.com"

Step 2: Configure IAM Role for OIDC

The IAM role that GitHub Actions will assume must have a trust policy that permits sts:AssumeRoleWithWebIdentity. This policy specifies which OIDC provider (GitHub Actions) and conditions (e.g., repository, branch, environment) are allowed to assume the role.

Create a file named trust-policy.json with the following content. Replace <YOUR_AWS_ACCOUNT_ID>, <YOUR_GITHUB_ORG>, and <YOUR_REPO_NAME> with your specific details. For branch-specific deployments, you can use ref:refs/heads/main or ref:refs/heads/*.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/token.actions.githubusercontent.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com", "token.actions.githubusercontent.com:sub": "repo:/:ref:refs/heads/main" } } } ] }

Important: The sub condition is critical. It must exactly match the expected subject provided by GitHub's OIDC token. For environment-specific deployments, it could be repo:<ORG>/<REPO>:environment:<ENVIRONMENT_NAME>. For any branch, use repo:<ORG>/<REPO>:ref:refs/heads/*.

Now, create or update your IAM role:

# Create a new role aws iam create-role \ --role-name github-actions-eks-deployer-role \ --assume-role-policy-document file://trust-policy.json # If the role already exists, update its trust policy aws iam update-assume-role-policy \ --role-name github-actions-eks-deployer-role \ --policy-document file://trust-policy.json

Step 3: Attach Permissions to IAM Role

The github-actions-eks-deployer-role needs permissions to interact with EKS and potentially other AWS services your deployment requires (e.g., ECR, S3). For EKS deployments, you'll typically need policies that allow cluster description, kubeconfig updates, and potentially full Kubernetes API access via kubectl.

Attach AWS managed policies or your custom policies. For EKS, AmazonEKSClusterPolicy and AmazonEKSAccessPolicy (or more granular custom policies) are common starting points.

aws iam attach-role-policy \ --role-name github-actions-eks-deployer-role \ --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy aws iam attach-role-policy \ --role-name github-actions-eks-deployer-role \ --policy-arn arn:aws:iam::aws:policy/AmazonEKSAccessPolicy # Example for ECR access if your deployment pulls images from ECR aws iam attach-role-policy \ --role-name github-actions-eks-deployer-role \ --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Crucial EKS-specific step: You must map this new IAM role to an EKS Kubernetes user or group within your EKS cluster's aws-auth ConfigMap. This grants the assumed IAM role the necessary Kubernetes RBAC permissions.

# First, get your current aws-auth configmap (if it exists) kubectl get -n kube-system configmap aws-auth -o yaml > aws-auth-current.yaml # Edit the aws-auth-current.yaml file. Add your role under mapRoles: # For example: # - rolearn: arn:aws:iam:::role/github-actions-eks-deployer-role # username: github-actions-user # groups: # - system:masters # Or a more specific group like 'edit', 'view', or custom RBAC roles # Apply the updated configmap kubectl apply -f aws-auth-current.yaml

It's recommended to assign the role to a less privileged group than system:masters for production environments. You can create custom ClusterRoles and ClusterRoleBindings in Kubernetes to define specific permissions.

Step 4: Update GitHub Actions Workflow

Modify your GitHub Actions workflow file (e.g., .github/workflows/deploy.yml) to use the OIDC-based role assumption.

name: EKS Deployment with OIDC on: push: branches: - main permissions: id-token: write # This is crucial for OIDC to work contents: read jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam:::role/github-actions-eks-deployer-role aws-region: YOUR_AWS_REGION # e.g., us-east-1 # role-session-name: github-actions-session # Optional, for easier CloudTrail auditing - name: Update Kubeconfig run: aws eks update-kubeconfig --name YOUR_EKS_CLUSTER_NAME --region YOUR_AWS_REGION - name: Deploy to EKS run: | kubectl apply -f k8s/deployment.yaml kubectl apply -f k8s/service.yaml kubectl rollout status deployment/your-app-deployment

Ensure permissions: id-token: write is present at the job or workflow level. The aws-actions/configure-aws-credentials action handles fetching the OIDC token and assuming the specified role.

Step 5: Test and Validate

Push changes to your repository to trigger the GitHub Actions workflow. Monitor the workflow logs carefully for any errors. If issues persist, refer to the troubleshooting checklist below.

Best Practices for Prevention & Performance Optimization

  • Principle of Least Privilege: Always grant the absolute minimum permissions required for the IAM role. Instead of system:masters in EKS, create custom Kubernetes ClusterRoles and bind them to the OIDC-assumed IAM role.
  • Strict OIDC Trust Policy Conditions: Use specific conditions in your IAM role's trust policy (e.g., repo:<ORG>/<REPO>:ref:refs/heads/main or environment:<ENVIRONMENT_NAME>) to limit who can assume the role. Avoid broad wildcards where possible.
  • Version Pinning for Actions: Always pin your GitHub Actions to a specific commit SHA (e.g., actions/checkout@v4.1.1 or actions/checkout@v4.1.1 with SHA) rather than just a major version (@v4) to ensure consistent behavior and prevent unexpected breaking changes.
  • Environment Variables for Secrets: Store sensitive values like EKS cluster names or account IDs as GitHub Secrets and reference them in your workflow. Avoid hardcoding.
  • Comprehensive Logging and Monitoring: Enable AWS CloudTrail for auditing IAM role assumption events. Use GitHub Actions' logging capabilities to debug workflow failures effectively. Integrate EKS logs with CloudWatch for application-level debugging.
  • Regular Audits: Periodically review your IAM roles, policies, and GitHub Actions workflows to ensure security and compliance. Remove any outdated or unnecessary permissions.
  • EKS aws-auth ConfigMap Management: Consider managing your aws-auth ConfigMap via Infrastructure as Code (IaC) tools like Terraform or CloudFormation to ensure consistency and prevent manual errors.

Frequently Asked Questions (FAQ)

Q1: Why am I getting "AccessDenied" even after configuring OIDC and the IAM role?

A1: This is typically due to one of two main reasons:

  1. Incorrect IAM Role Permissions: The IAM role assumed by GitHub Actions lacks the necessary permissions to perform the AWS API calls (e.g., eks:UpdateKubeconfig, ecr:GetAuthorizationToken) or Kubernetes API calls. Double-check the attached IAM policies.
  2. EKS aws-auth ConfigMap: Even if you assume the IAM role successfully, that role needs to be mapped in your EKS cluster's aws-auth ConfigMap to a Kubernetes user/group with sufficient RBAC permissions. If this mapping is missing or incorrect, kubectl commands will fail with authorization errors.
Review both the IAM policies attached to your role and the EKS aws-auth ConfigMap for the assumed role.

Q2: How do I debug OIDC role assumption locally or test the OIDC token?

A2: Debugging OIDC tokens directly can be complex as they are short-lived. However, you can use the AWS CLI to test role assumption with a dummy OIDC token for diagnostic purposes.

  • First, you would typically need to get an OIDC token from a valid OIDC provider. For GitHub Actions, this token is automatically generated.
  • For local testing, you can try to craft a minimal JWT token (e.g., using jwt.io with a dummy signature) and then attempt to assume the role using aws sts assume-role-with-web-identity. This is more advanced and often not straightforward due to signature validation.
  • A more practical approach is to ensure your IAM role's trust policy has correct conditions, then trigger the GitHub Actions workflow and examine the detailed logs. CloudTrail logs will also show if the AssumeRoleWithWebIdentity call was made and if it failed due to trust policy issues.
Focus on validating the OIDC provider's URL, the aud (audience), and sub (subject) values in your IAM role's trust policy against what GitHub Actions generates.

Q3: Is it safe to use OIDC with EKS? What are the security benefits?

A3: Yes, using OIDC for GitHub Actions deployments to EKS is highly recommended and considered a significant security best practice. The primary benefits include:

  • No Long-Lived Credentials: Eliminates the need to store static AWS access keys and secret keys in GitHub Secrets, which are susceptible to compromise. OIDC tokens are short-lived and automatically exchanged for temporary AWS credentials.
  • Fine-Grained Access Control: IAM role trust policies can be configured with highly specific conditions (repository, branch, environment, workflow file path) to limit who can assume the role, reducing the blast radius in case of a token leak.
  • Improved Auditability: All role assumption events are logged in AWS CloudTrail, providing a clear audit trail of who (which GitHub repo/workflow) accessed your AWS resources.
  • Reduced Operational Overhead: No need for manual rotation of static credentials.
This approach significantly reduces the attack surface compared to traditional methods using static AWS credentials.

Popular posts from this blog

Debugging ImagePullBackOff in Kubernetes EKS with AWS ECR authentication issues

Debugging ImagePullBackOff in Kubernetes EKS with AWS ECR Authentication Issues Encountering ImagePullBackOff is a common hurdle for developers managing containerized applications on Kubernetes, especially within AWS EKS when images are stored in Amazon Elastic Container Registry (ECR). This error signifies that Kubernetes could not pull a required container image, frequently stemming from underlying authentication and authorization problems with ECR. A robust cloud hosting server environment like EKS demands precise configuration for seamless operation. This guide will walk through diagnosing and resolving these authentication issues, ensuring your applications deploy smoothly on your scalable cloud infrastructure . Brief Introduction & Symptom Analysis The ImagePullBackOff status indicates that a Kubernetes Pod is repeatedly failing to pull an image. When combined with AWS ECR, this often points to EKS worker nodes lacking the necess...
Read more

Fixing EKS Pod CrashLoopBackOff Due to Readiness Probe Failures

Tech Note: Always backup your configuration files before applying any changes to production environments. Fixing EKS Pod CrashLoopBackOff Due to Readiness Probe Failures In modern cloud-native architectures, Kubernetes (K8s) is the de-facto standard for container orchestration, with Amazon Elastic Kubernetes Service (EKS) being a popular choice for managed K8s on AWS. However, even in highly resilient environments, pods can sometimes enter a CrashLoopBackOff state. One of the most common and often misunderstood reasons for this is a failing readiness probe . This comprehensive guide, crafted by a Senior Cloud Solution Architect and Software Engineer, will demystify readiness probe failures on EKS, provide a structured troubleshooting methodology, and offer best practices for prevention. Symptom Analysis & Root Causes Understanding the symptoms and underlying causes is the first critical step in resolving any complex issue. A CrashLoopBackOff state in...
Read more

Resolve Nginx `upstream prematurely closed connection` with SSL termination for Docker containers

Resolve Nginx `upstream prematurely closed connection` with SSL termination for Docker containers Encountering the Nginx error `upstream prematurely closed connection` when running applications within Docker containers, especially with SSL termination configured, can be a frustrating experience for any Cloud Engineer or developer. This guide provides a comprehensive, step-by-step approach to diagnose and resolve this common issue, ensuring your services remain robust and highly available on any cloud hosting server . Understanding and mitigating this error is crucial for maintaining a reliable scalable cloud infrastructure and an efficient VPS server management strategy. Brief Introduction & Symptom Analysis The `upstream prematurely closed connection` error indicates that Nginx, acting as a reverse proxy, initiated a connection to an upstream server (your Dockerized application) but the upstream closed the connection before Nginx expec...
Read more