Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing Kubernetes ImagePullBackOff from Private ECR on AWS EKS with IRSA

The ImagePullBackOff error is a common nuisance for Kubernetes users, signaling that the Kubelet couldn't pull a container image. When operating on AWS EKS and pulling images from a private Amazon Elastic Container Registry (ECR) using IAM Roles for Service Accounts (IRSA), this error often points to misconfigurations in permissions, trust policies, or network access. This comprehensive guide will walk you through diagnosing and resolving this issue, ensuring your workloads can successfully fetch images.

Symptom Analysis & Root Causes

The ImagePullBackOff status typically means that Kubernetes attempted to pull an image multiple times and failed. To understand the underlying cause, we need to inspect the pod events. When dealing with private ECR on EKS with IRSA, the common culprits include:

• Incorrect IAM Role for Service Account (IRSA) Configuration: The most frequent cause. This could mean:
  • The Kubernetes Service Account (KSA) does not have the correct eks.amazonaws.com/role-arn annotation.
  • The associated AWS IAM Role's trust policy does not correctly allow the EKS OIDC provider to assume the role.
  • The IAM Role lacks the necessary permissions to access ECR (e.g., ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:BatchGetImage).
• Missing or Incorrect Service Account in Pod Spec: The pod might not be explicitly configured to use the KSA that is linked to the IAM Role.
• ECR Repository Policy Issues: The ECR repository policy might restrict access, preventing the IAM role from pulling images, especially in cross-account scenarios.
• Network Connectivity Problems:
  • EKS worker node security groups do not allow outbound HTTPS traffic (port 443) to ECR public endpoints or ECR VPC endpoints.
  • Missing or misconfigured VPC endpoints for ECR in private subnets.
  • Network ACLs blocking traffic.
• Incorrect Image URI: A typo in the ECR image URI or a non-existent image tag.
• AWS CLI/Kubectl Version Mismatch: Outdated tools can sometimes lead to authentication issues, though less common with IRSA.

Step-by-Step Resolution Guide

Follow these steps to systematically diagnose and resolve ImagePullBackOff errors originating from ECR with IRSA on AWS EKS.

Prerequisites:

• AWS CLI installed and configured.
• kubectl installed and configured to connect to your EKS cluster.
• eksctl installed (highly recommended for managing EKS clusters and IRSA).

Step 1: Verify Pod Status and Events

Start by examining the problematic pod to understand why the image pull failed.

kubectl get pods -n <your-namespace> kubectl describe pod <pod-name> -n <your-namespace>

Look for events like Failed to pull image, Error response from daemon, or no basic auth credentials. These messages provide crucial hints.

Step 2: Inspect Kubernetes Service Account (KSA) Configuration

Ensure your pod is configured to use an existing KSA and that this KSA is correctly annotated for IRSA.

2.1. Check Pod Spec for serviceAccountName:

Verify that your Pod or Deployment specification includes serviceAccountName.

# Example Deployment snippet apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: spec: serviceAccountName: my-service-account # <--- THIS MUST BE PRESENT containers: - name: my-container image: <your-ecr-repo-uri>/<image-name>:<tag>

If serviceAccountName is missing, add it and redeploy. If you don't have a KSA, create one:

kubectl create serviceaccount my-service-account -n <your-namespace>

2.2. Verify KSA Annotation:

The KSA must be annotated with the ARN of the AWS IAM Role it should assume.

kubectl describe serviceaccount my-service-account -n <your-namespace>

Look for an annotation like eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-ecr-access-role. If it's missing or incorrect, add/update it:

kubectl annotate serviceaccount my-service-account -n <your-namespace> eks.amazonaws.com/role-arn="arn:aws:iam::<AWS_ACCOUNT_ID>:role/<your-iam-role-name>" --overwrite

Step 3: Validate AWS IAM Role Configuration

The IAM Role associated with your KSA needs the correct trust policy and permissions.

3.1. Verify OIDC Provider Association:

Ensure your EKS cluster has an OIDC Identity Provider associated.

eksctl utils associate-iam-oidc-provider --cluster <your-cluster-name> --approve

This command ensures the OIDC provider URL is registered with IAM.

3.2. Inspect IAM Role Trust Policy:

The IAM Role's trust policy must allow the EKS OIDC provider to assume it. Go to IAM Console -> Roles -> <your-iam-role-name> -> Trust relationships tab. It should look like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<your-namespace>:my-service-account" } } } ] }

Ensure <AWS_ACCOUNT_ID>, <AWS_REGION>, <OIDC_ID>, <your-namespace>, and my-service-account match your configuration. The <OIDC_ID> can be found from your EKS cluster details in the AWS console.

3.3. Verify IAM Role Permissions Policy:

The IAM Role needs permissions to pull images from ECR. Attach a policy like the following:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage" ], "Resource": "*" } ] }

For stricter least privilege, replace "Resource": "*" with specific ECR repository ARNs: "Resource": "arn:aws:ecr:<AWS_REGION>:<AWS_ACCOUNT_ID>:repository/<your-repository-name>".

Step 4: Validate ECR Repository Policy

If you're pulling from a cross-account ECR repository, or if the repository has a custom policy, ensure it explicitly allows your IAM role. Go to ECR Console -> Repositories -> <your-repository> -> Permissions tab.

{ "Version": "2008-10-17", "Statement": [ { "Sid": "AllowEKSImagePull", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<your-iam-role-name>" }, "Action": [ "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:GetRepositoryPolicy", "ecr:DescribeRepositories", "ecr:ListImages", "ecr:DescribeImages", "ecr:BatchGetImage" ] } ] }

This policy allows the IAM role to pull images.

Step 5: Check Network Connectivity

Network misconfigurations can prevent worker nodes from reaching ECR.

5.1. EKS Worker Node Security Groups:

Ensure the security groups attached to your EKS worker nodes allow outbound HTTPS (port 443) traffic. If using public ECR endpoints, the destination would be the internet. If using VPC endpoints, the destination would be the security group of the VPC endpoint.

5.2. VPC Endpoints for ECR:

If your EKS cluster operates in private subnets, you need VPC endpoints for ECR to pull images without traversing the internet. You'll need two endpoints:

• com.amazonaws.<AWS_REGION>.ecr.dkr (for Docker image operations)
• com.amazonaws.<AWS_REGION>.s3 (ECR uses S3 for storing image layers)
• com.amazonaws.<AWS_REGION>.ecr.api (for ECR API calls, optional but recommended)

Verify that these endpoints are correctly configured for the VPC and subnets where your EKS worker nodes reside, and that their security groups allow inbound HTTPS from your worker node security groups.

Step 6: Confirm Image URI and Tag

A simple but often overlooked issue is an incorrect image URI or tag.

aws ecr describe-images --repository-name <your-repository-name> --query 'imageDetails[].imageTags[]'

Cross-reference the output with the image URI in your pod specification.

Best Practices for Prevention & Performance Optimization

Adopting these practices can prevent future ImagePullBackOff issues and optimize your EKS image pulling workflow.

• Automate IRSA Setup: Use tools like eksctl or AWS CloudFormation/Terraform to manage OIDC provider and IAM role creation for consistent and error-free IRSA setup.
• Least Privilege Principle: Always grant the minimum necessary ECR permissions to your IAM roles. Avoid "Resource": "*" where specific repository ARNs can be used.
• Use Image Digests: Instead of tags (e.g., latest), specify images by their immutable digest (e.g., <image>@sha256:<digest>). This prevents unexpected image changes and enhances security.
• ECR Lifecycle Policies: Implement lifecycle policies in ECR to automatically clean up old, unused image versions, reducing storage costs and maintaining registry hygiene.
• Monitor EKS and ECR Logs: Integrate EKS control plane logs with CloudWatch and monitor ECR events for unauthorized access attempts or pull failures.
• Keep Kubelet and AWS CLI Updated: Ensure your worker nodes' Kubelet version and local AWS CLI are up-to-date to benefit from the latest features and bug fixes related to IAM authentication.
• Health Checks: Implement proper liveness and readiness probes in your pod definitions to ensure applications are healthy and reachable after image pulls.

Frequently Asked Questions (FAQs)

Q1: Why should I use IRSA instead of attaching an IAM role to my EKS worker node instance profile?

A: IRSA (IAM Roles for Service Accounts) provides fine-grained permissions at the pod level, adhering to the principle of least privilege. When an IAM role is attached to the worker node instance profile, all pods on that node inherit those permissions, potentially granting more access than needed. IRSA allows you to specify a unique IAM role for each Kubernetes Service Account, which pods can then assume, significantly improving security posture and reducing the blast radius in case of a compromise.

Q2: My pod events show "no basic auth credentials". What does this mean in the context of ECR and IRSA?

A: The "no basic auth credentials" message indicates that the Kubelet attempted to pull an image but did not have the necessary authentication token. With ECR and IRSA, this almost always points to an issue with the IAM Role's permissions or trust policy, or the KSA's annotation. Specifically, the IAM role associated with the KSA likely lacks the ecr:GetAuthorizationToken permission, which is required for fetching temporary ECR login credentials. It could also mean the KSA is not correctly linked to the IAM role, or the pod isn't using the annotated KSA.

Q3: Can I pull images from a private ECR repository in a different AWS account using IRSA?

A: Yes, you can. To achieve this, the IAM Role used by your EKS service account (in Account A) needs to have a policy allowing ECR permissions. Additionally, the ECR repository itself (in Account B) must have a resource-based policy that explicitly allows the IAM Role from Account A to perform ECR actions like ecr:BatchGetImage and ecr:BatchCheckLayerAvailability. Ensure both the IAM role's permissions policy and the ECR repository's resource policy are correctly configured for cross-account access.