Troubleshooting Kubernetes ImagePullBackOff from Private ECR on AWS EKS with IRSA

Tech Note: Always backup your configuration files before applying any changes to production environments.

Troubleshooting Kubernetes ImagePullBackOff from Private ECR on AWS EKS with IRSA

The ImagePullBackOff error is a common frustration for Kubernetes users, especially when working with private container registries like Amazon Elastic Container Registry (ECR) on AWS Elastic Kubernetes Service (EKS) using IAM Roles for Service Accounts (IRSA). This comprehensive guide provides senior cloud solution architects and software engineers with a detailed understanding of the problem, its root causes, and a systematic, step-by-step troubleshooting manual to resolve it, ensuring your applications deploy smoothly.

Understanding ImagePullBackOff and IRSA

When a Kubernetes pod fails to start with an ImagePullBackOff status, it means the Kubelet on the worker node was unable to pull the specified container image from its registry. While this can happen for various reasons (incorrect image name, network issues), the complexity often increases when dealing with private registries like ECR and authentication mechanisms like IRSA.

IRSA (IAM Roles for Service Accounts) is AWS's recommended method for granting AWS permissions to pods running on EKS. Instead of granting permissions to the worker node's instance profile (which would grant all pods on that node the same permissions), IRSA allows you to associate an IAM role with a Kubernetes Service Account. Pods configured to use that Service Account can then assume the associated IAM role and gain the specific AWS permissions defined in it. For ECR access, this means the pod needs permissions to authenticate with ECR and pull images.

Symptom Analysis & Root Causes

Identifying the Symptoms

The primary symptom is a pod stuck in a Pending or CrashLoopBackOff state, with the output of kubectl get pods showing ImagePullBackOff or ErrImagePull. Deeper inspection using kubectl describe pod will reveal the specific error.

kubectl get pods # Example Output: # NAME READY STATUS RESTARTS AGE # my-app-5f9c6d8d4-abcde 0/1 ImagePullBackOff 0 2m kubectl describe pod my-app-5f9c6d8d4-abcde # Look for events similar to: # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal Scheduled 2m default-scheduler Successfully assigned default/my-app-5f9c6d8d4-abcde to ip-192-168-10-10.ec2.internal # Normal Pulling 50s (x3 over 2m) kubelet Pulling image "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest" # Warning Failed 49s (x3 over 2m) kubelet Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://123456789012.dkr.ecr.us-east-1.amazonaws.com/v2/my-repo/manifests/latest": no basic auth credentials # Warning Failed 49s (x3 over 2m) kubelet Error: ImagePullBackOff

Common Root Causes with IRSA and ECR

• IAM Role Missing ECR Permissions: The most frequent cause. The IAM role associated with the Service Account lacks the necessary permissions to authenticate with ECR and pull images. Required permissions typically include:
  • ecr:GetAuthorizationToken
  • ecr:BatchCheckLayerAvailability
  • ecr:GetDownloadUrlForLayer
  • ecr:BatchGetImage
• IAM Role Trust Policy Misconfiguration: The IAM role's trust policy does not permit the EKS cluster's OIDC provider to assume the role. The policy must explicitly allow sts:AssumeRoleWithWebIdentity for the OIDC provider associated with your EKS cluster.
• Service Account Annotation Missing or Incorrect: The Kubernetes Service Account must be correctly annotated with the IAM role ARN (eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/YOUR_IAM_ROLE_NAME).
• Pod Spec Not Using Correct Service Account: The pod's definition might not specify the correct serviceAccountName, or it might be defaulting to the default service account which lacks the necessary IRSA configuration.
• OIDC Provider Not Configured for EKS Cluster: The OIDC Identity Provider must be created and associated with your EKS cluster in AWS IAM. If this is missing, IRSA will not function.
• ECR Repository Policy Restrictions: The ECR repository itself might have a policy that denies access to the specific IAM role or account, even if the role's permissions are correct.
• Network Connectivity Issues: The EKS worker nodes might not have outbound connectivity to the ECR endpoints. This can be due to overly restrictive Security Groups, Network ACLs, or missing VPC Endpoints for ECR if the nodes are in private subnets.
• Incorrect Image Name or Tag: A simple typo in the image path or tag can also lead to this error. Ensure the full ECR repository URI and tag are accurate.

Step-by-Step Resolution Guide

Follow these steps systematically to diagnose and resolve your ImagePullBackOff issue.

Step 1: Verify the Pod and Error Details

Confirm the error is ImagePullBackOff and examine the specific message. This provides clues about whether it's an authentication, authorization, or network issue.

kubectl get pods --all-namespaces -o wide # Identify the pod in ImagePullBackOff status and its namespace. kubectl describe pod -n # Look for "Events:" section at the bottom. # Pay close attention to "Failed to pull image" or "Error: ImagePullBackOff" messages. # Common errors: "no basic auth credentials", "unauthorized: authentication required", "i/o timeout".

Step 2: Check Pod's Service Account and IRSA Annotation

Ensure the pod is configured to use the correct service account, and that service account has the IRSA annotation pointing to the intended IAM role.

# Get the service account used by the pod kubectl get pod -n -o yaml | grep serviceAccountName # Example output: serviceAccountName: my-ecr-sa # Get details of the service account kubectl get sa -n -o yaml # Look for annotations: # annotations: # eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/MyEKSImagePullRole # # ... other annotations # If the annotation is missing or incorrect, add/correct it and recreate the pod.

If the Service Account is missing or incorrect, update your deployment manifest and reapply:

# Example ServiceAccount definition apiVersion: v1 kind: ServiceAccount metadata: name: my-ecr-sa namespace: my-app-namespace annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/MyEKSImagePullRole --- # Example Deployment referencing the ServiceAccount apiVersion: apps/v1 kind: Deployment metadata: name: my-app namespace: my-app-namespace spec: selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: serviceAccountName: my-ecr-sa # <--- Ensure this matches the ServiceAccount name containers: - name: my-app-container image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest

Step 3: Validate IAM Role Permissions and Trust Policy

Navigate to the AWS IAM console or use the AWS CLI to inspect the IAM role linked to your Service Account.

# Use AWS CLI to get the role details aws iam get-role --role-name MyEKSImagePullRole # Check the "AssumeRolePolicyDocument" for the trust policy. # It should contain something like: # { # "Version": "2012-10-17", # "Statement": [ # { # "Effect": "Allow", # "Principal": { # "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED99EE863F6964FXXXXXXX" # }, # "Action": "sts:AssumeRoleWithWebIdentity", # "Condition": { # "StringEquals": { # "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED99EE863F6964FXXXXXXX:sub": "system:serviceaccount:my-app-namespace:my-ecr-sa" # } # } # } # ] # }

Verify the OIDC provider ARN matches your EKS cluster's OIDC provider (see Step 4) and the sub condition matches your namespace and service account name.

Next, check the IAM role's permissions. It must have an attached policy (or inline policy) granting the necessary ECR permissions:

• ecr:GetAuthorizationToken
• ecr:BatchCheckLayerAvailability
• ecr:GetDownloadUrlForLayer
• ecr:BatchGetImage

A common AWS managed policy for this is AmazonEC2ContainerRegistryReadOnly or a custom policy like:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:GetAuthorizationToken" ], "Resource": "*" } ] }

Step 4: Confirm EKS OIDC Provider Configuration

Ensure your EKS cluster has an associated OIDC identity provider in IAM.

# Get your cluster's OIDC issuer URL aws eks describe-cluster --name --query "cluster.identity.oidc.issuer" --output text # Example output: https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED99EE863F6964FXXXXXXX # Then, check if this OIDC provider exists in IAM: aws iam list-open-id-connect-providers | grep EXAMPLED99EE863F6964FXXXXXXX # The output should contain the ARN of your OIDC provider. # If not present, you need to create it (eksctl or AWS console can do this).

Step 5: Check ECR Repository Policy (if applicable)

If you have specific repository policies, ensure they don't explicitly deny access to the IAM role.

aws ecr get-repository-policy --repository-name my-repo # Review the policy JSON for any Deny statements that might affect your IAM role.

Step 6: Network Connectivity to ECR

If your EKS worker nodes are in private subnets, you typically need VPC Endpoints for ECR. Even in public subnets, Security Groups and Network ACLs could block access.

• VPC Endpoints: Ensure VPC Endpoints for ECR are correctly configured for your VPC (ecr.dkr and ecr.api).
• Security Groups: The Security Group attached to your EKS worker nodes must allow outbound HTTPS (port 443) traffic to ECR. If using VPC endpoints, it must allow traffic to the endpoint's security group.
• Network ACLs: Verify Network ACLs for subnets hosting worker nodes allow inbound/outbound on port 443.

You can test connectivity from an EKS worker node (if SSH access is available or via a debug pod):

# From a worker node or a debug pod: # Install curl if not present sudo apt-get update && sudo apt-get install -y curl # for Ubuntu/Debian sudo yum install -y curl # for Amazon Linux/CentOS # Test connectivity to ECR API endpoint curl -v https://ecr.us-east-1.amazonaws.com/ # Expected: HTTP/1.1 200 OK or similar authentication challenge. # Not expected: connection refused, timeout, or DNS resolution failure. # Test connectivity to ECR registry endpoint (replace with your region and account) curl -v https://123456789012.dkr.ecr.us-east-1.amazonaws.com/v2/ # Expected: HTTP/1.1 401 Unauthorized or 200 OK (if already authenticated). # Not expected: connection refused, timeout, or DNS resolution failure.

Step 7: Recreate the Pod

After making any changes (IAM role, Service Account, network configuration), delete the problematic pod(s) to force Kubernetes to schedule new ones with the updated configuration.

kubectl delete pod -n # Or for deployments: kubectl rollout restart deployment -n

Best Practices for Prevention & Performance Optimization

• Automate IRSA Setup: Use tools like eksctl, AWS CDK, or Terraform to provision your EKS cluster and IRSA roles. This reduces manual errors and ensures consistency.
• Principle of Least Privilege: Grant your IRSA roles only the minimum necessary ECR permissions (GetAuthorizationToken, BatchCheckLayerAvailability, GetDownloadUrlForLayer, BatchGetImage) and specific repository access if possible, instead of "Resource": "*".
• Use VPC Endpoints for ECR: For private subnets, configure VPC Endpoints for ECR (ecr.dkr and ecr.api) to ensure secure, reliable, and performant image pulls without traversing the public internet.
• Consistent Image Naming & Tagging: Enforce strict naming conventions for your ECR repositories and container image tags. Avoid :latest in production; use specific, immutable tags.
• Version Control for Kubernetes Manifests: Store all your Kubernetes manifests (Deployments, ServiceAccounts, etc.) in a version control system (e.g., Git) and use CI/CD pipelines for deployment.
• Regularly Review IAM Policies: Periodically audit your IAM roles and policies to ensure they align with current requirements and best security practices.
• Monitor EKS & ECR Logs: Integrate EKS and ECR with CloudWatch Logs to gain deeper insights into authentication and pull failures, enabling quicker debugging.

Frequently Asked Questions (FAQs)

Q1: What exactly is ImagePullBackOff and why does it happen with private registries?

A: ImagePullBackOff is a Kubernetes status indicating that a pod could not pull its required container image. It often happens with private registries like ECR because the Kubernetes worker node (or more specifically, the kubelet agent) needs valid credentials to authenticate with the private registry. Without correct authentication (e.g., missing IAM permissions for IRSA, incorrect IAM role, or network issues), the pull operation fails, and Kubernetes repeatedly tries, resulting in the "BackOff" behavior.

Q2: Do I still need imagePullSecrets if I'm using IRSA for ECR access?

A: No, if IRSA is correctly configured for ECR access, you do not need imagePullSecrets. IRSA works by leveraging a temporary, short-lived token obtained via AWS STS (sts:AssumeRoleWithWebIdentity) that the kubelet uses to authenticate with ECR. This mechanism securely provides credentials directly to the pod's service account, abstracting away the need for manually managed secrets for registry authentication.

Q3: How can I efficiently test ECR access from an EKS worker node to rule out network issues?

A: The most direct way is to SSH into an EKS worker node (if allowed) and manually attempt to log in to ECR using the AWS CLI configured with the node's instance profile (which has temporary credentials). However, to specifically test the IRSA role, a more accurate method is to deploy a temporary debug pod configured with the problematic Service Account and then execute commands inside that pod:

# 1. Create a debug pod using the service account in question cat < spec: serviceAccountName: containers: - name: aws-cli image: amazon/aws-cli:latest command: ["tail", "-f", "/dev/null"] # Keep pod running imagePullPolicy: Always # Ensure it tries to pull EOF # 2. Wait for the pod to be running (if it can pull the aws-cli image) # Then execute into the pod kubectl exec -it ecr-debug-pod -n -- /bin/bash # 3. Inside the pod, attempt to get ECR login credentials # Replace and with your specifics aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com # If this command succeeds, it confirms that the IRSA role (via the service account) # has the necessary permissions and network connectivity to ECR. # If it fails, the error message will help pinpoint the exact problem (e.g., access denied, network timeout). # 4. Clean up the debug pod kubectl delete pod ecr-debug-pod -n