Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing Kubernetes ImagePullBackOff for Private ECR Repositories on AWS EKS with IRSA

As a Senior Cloud Solution Architect and Software Engineer, I frequently encounter challenges in deploying containerized applications on Kubernetes. One of the most common and frustrating issues for users leveraging AWS Elastic Kubernetes Service (EKS) with private Elastic Container Registry (ECR) repositories is the ImagePullBackOff error. This guide provides a comprehensive technical breakdown and a step-by-step troubleshooting manual, focusing on the highly recommended approach of using IAM Roles for Service Accounts (IRSA).

Understanding ImagePullBackOff on EKS with ECR and IRSA

The ImagePullBackOff status in Kubernetes indicates that a pod failed to pull its required container image. While this error can stem from various causes, when working with private AWS ECR repositories on EKS and relying on IRSA for authentication, the problem almost invariably points to an access or configuration issue related to IAM, networking, or the Kubernetes Service Account itself. IRSA is AWS's recommended approach for granting IAM permissions to Kubernetes pods, offering fine-grained access control and enhanced security over traditional node instance profiles.

Symptom Analysis & Root Causes

When your pods are stuck in an ImagePullBackOff state, you'll typically see:

• Pod Status: Running kubectl get pods will show your pod in a Pending state, often with a STATUS of ImagePullBackOff or ErrImagePull.
• Events Log: Running kubectl describe pod <pod-name> -n <namespace> will reveal events like:
  • Failed to pull image "xxxxxxxxxxxx.dkr.ecr.region.amazonaws.com/my-repo/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://xxxxxxxxxxxx.dkr.ecr.region.amazonaws.com/v2/my-repo/my-image/manifests/latest": no basic auth credentials
  • Failed to pull image "xxxxxxxxxxxx.dkr.ecr.region.amazonaws.com/my-repo/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: Head "https://xxxxxxxxxxxx.dkr.ecr.region.amazonaws.com/v2/my-repo/my-image/manifests/latest": dial tcp x.x.x.x:443: connect: connection refused

Common root causes for this specific scenario (EKS, Private ECR, IRSA) include:

• Incorrect IRSA Setup:
  • OIDC Provider Mismatch: The IAM OIDC provider for your EKS cluster is not correctly configured or associated.
  • IAM Role Trust Policy: The IAM role linked to your Kubernetes Service Account does not have the correct trust policy, specifically allowing sts:AssumeRoleWithWebIdentity from your EKS OIDC provider.
  • Service Account Annotation Missing/Incorrect: The Kubernetes Service Account used by your deployment is not correctly annotated with eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<your-irsa-role>.
• Insufficient IAM Permissions: The IAM role assumed by the Kubernetes Service Account lacks the necessary permissions to interact with ECR. Key permissions include ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage.
• ECR Repository Policy: If a custom repository policy is applied to the ECR repository, it might explicitly deny access to the IAM role.
• Network Connectivity Issues:
  • VPC Endpoints (Private Subnets): If your worker nodes are in private subnets, you must have VPC Interface Endpoints configured for ECR (com.amazonaws.region.ecr.api, com.amazonaws.region.ecr.dkr) and S3 (com.amazonaws.region.s3) to allow image pulls without traversing the public internet.
  • Security Groups/Network ACLs: Restrictive Security Groups on worker nodes or VPC Endpoints, or Network ACLs, preventing outbound HTTPS (port 443) traffic to ECR.
  • DNS Resolution: Issues with DNS resolution for ECR endpoints within the VPC.
• Incorrect Image Name/Tag: The image name or tag specified in the pod definition is incorrect or doesn't exist in the ECR repository.

Step-by-Step Resolution Guide

Follow these steps to diagnose and fix ImagePullBackOff errors with private ECR repositories on EKS using IRSA.

Prerequisites:

• AWS CLI (configured with appropriate credentials and region)
• kubectl (configured to connect to your EKS cluster)
• eksctl (recommended for OIDC provider management)

Step 1: Verify Pod Status and Events

First, confirm the ImagePullBackOff status and check the detailed events to get initial clues.

kubectl get pods -n <your-namespace> kubectl describe pod <pod-name> -n <your-namespace>

Look for messages indicating authentication failures, network issues, or image not found errors.

Step 2: Check OIDC Provider Association

An IAM OIDC provider must be associated with your EKS cluster to enable IRSA. Use eksctl to check and create/associate if missing.

# Check if OIDC provider exists eksctl utils associate-iam-oidc-provider --cluster <your-cluster-name> --approve --region <your-region> # If it indicates that it will create one, proceed. Otherwise, it's already there.

You can also verify this in the AWS Console under IAM -> Identity Providers, where you should see a provider with a URL matching your EKS cluster's OIDC issuer URL (e.g., https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>).

Step 3: Inspect Kubernetes Service Account (SA)

Your pod must use a Service Account, and that SA must be annotated with the IAM Role ARN.

kubectl get sa <your-service-account-name> -n <your-namespace> -o yaml

Ensure the output contains an annotation like this:

apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<your-account-id>:role/<your-irsa-role-name> # ... other annotations name: <your-service-account-name> namespace: <your-namespace>

Also, ensure your pod's deployment/definition references this Service Account:

apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: spec: serviceAccountName: <your-service-account-name> containers: - name: my-container image: <your-ecr-image>

If the Service Account or annotation is missing, update your Kubernetes manifest and re-apply.

Step 4: Verify IAM Role and Trust Policy

The IAM Role specified in the Service Account annotation must exist and have the correct trust policy.

aws iam get-role --role-name <your-irsa-role-name>

Check the AssumeRolePolicyDocument for the following structure. Replace <oidc-provider-url> with your EKS cluster's OIDC issuer URL (e.g., oidc.eks.us-east-1.amazonaws.com/id/ABCDEF1234567890) and <namespace>:<service-account-name> with your specific values.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<your-account-id>:oidc-provider/<oidc-provider-url>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "<oidc-provider-url>:sub": "system:serviceaccount:<namespace>:<service-account-name>" } } } ] }

If the trust policy is incorrect, update it using the AWS CLI or Console.

# Save the new trust policy to a file, e.g., trust-policy.json aws iam update-assume-role-policy --role-name <your-irsa-role-name> --policy-document file://trust-policy.json

Step 5: Check IAM Role Permissions for ECR Access

The IAM role must have permissions to authenticate with ECR and pull images. The managed policy AmazonEC2ContainerRegistryReadOnly is usually sufficient. If you use a custom policy, ensure it includes:

• ecr:GetAuthorizationToken
• ecr:BatchCheckLayerAvailability
• ecr:GetDownloadUrlForLayer
• ecr:BatchGetImage

{ "Version": "2012-10-17", "Statement": [ { "Sid": "ECRReadAccess", "Effect": "Allow", "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:GetAuthorizationToken" ], "Resource": "*" # You can restrict this to specific ECR repository ARNs if desired } ] }

Attach this policy (or AmazonEC2ContainerRegistryReadOnly) to the IAM role.

aws iam attach-role-policy --role-name <your-irsa-role-name> --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Step 6: Validate ECR Repository Policy (if custom)

If you have a specific repository policy on your ECR repository, ensure it doesn't explicitly deny access to your IAM role.

aws ecr get-repository-policy --repository-name <your-ecr-repo-name> --registry-id <your-account-id>

Review the policy for any deny statements that might override the role's permissions.

Step 7: Check Network Connectivity (VPC Endpoints, Security Groups)

ECR API: com.amazonaws.<region>.ecr.api

ECR DKR: com.amazonaws.<region>.ecr.dkr

S3: com.amazonaws.<region>.s3 (ECR uses S3 for image storage, so access to S3 is also required).

Ensure the Security Groups attached to your worker nodes and VPC Endpoints allow outbound HTTPS (port 443) traffic to the ECR/S3 endpoints. You can test connectivity from within a debug pod on the cluster:

kubectl run -it --rm --restart=Never debug-pod --image=alpine -- /bin/sh # Inside the pod: apk add curl curl -v https://<your-account-id>.dkr.ecr.<region>.amazonaws.com/v2/ # You should see a 401 Unauthorized response, not a connection refused. # Test ECR login (requires IRSA permissions on the debug pod's SA if used) apk add docker aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.<region>.amazonaws.com # Login Succeeded should appear. This confirms IRSA and network are working.

Step 8: Confirm Image Name and Tag

Double-check the image name and tag in your deployment manifest. A simple typo can cause ImagePullBackOff.

kubectl get deployment <your-deployment-name> -n <your-namespace> -o yaml | grep image: # Compare this with the actual image in ECR. aws ecr describe-images --repository-name <your-ecr-repo-name> --registry-id <your-account-id> --region <your-region>

Step 9: Re-deploy the Pod/Deployment

After making any changes (Service Account, IAM Role, network), roll out a new revision of your deployment to ensure the changes take effect.

kubectl rollout restart deployment <your-deployment-name> -n <your-namespace>

Best Practices for Prevention & Performance Optimization

• Automate IRSA Setup: Use tools like eksctl or Terraform to manage your EKS clusters, OIDC providers, IAM roles, and Service Accounts. This ensures consistency and reduces manual error.
• Least Privilege IAM Roles: Always grant only the necessary permissions to your IAM roles. For ECR image pulls, AmazonEC2ContainerRegistryReadOnly is ideal. Avoid overly permissive roles.
• Dedicated Service Accounts: Create specific Kubernetes Service Accounts for each application or microservice, each linked to a distinct IAM role with only the permissions that particular workload needs.
• VPC Endpoints for Private Networks: Always implement VPC Interface Endpoints for ECR (API and DKR) and S3 when running EKS worker nodes in private subnets. This not only resolves connectivity but also enhances security by keeping traffic within the AWS network.
• Centralized Logging and Monitoring: Integrate EKS with CloudWatch Logs and other monitoring tools. Detailed logs from Kubernetes events and ECR access logs can provide valuable insights during troubleshooting.
• ECR Lifecycle Policies: Implement ECR lifecycle policies to automatically clean up old or untagged images. This keeps your repositories lean and reduces storage costs.
• Image Scanning: Enable ECR image scanning to detect vulnerabilities early in the development pipeline, preventing problematic images from reaching production.

Frequently Asked Questions (FAQs)

Q1: Why should I use IRSA instead of attaching an IAM role to the EC2 instance profile of my worker nodes?

A1: IRSA offers significantly improved security and granularity. When you attach an IAM role to the EC2 instance profile, all pods running on that node inherit those permissions, creating a broad attack surface. With IRSA, each Kubernetes Service Account (and thus the pods using it) can assume a distinct IAM role with only the necessary permissions, adhering to the principle of least privilege. This greatly reduces the blast radius in case of a container compromise.

Q2: Do I still need imagePullSecrets if I'm using IRSA for ECR?

A2: No, generally not for ECR. IRSA is designed to eliminate the need for manually managing imagePullSecrets for ECR repositories. When a pod is configured with a Service Account annotated for IRSA, the EKS credential provider automatically handles fetching temporary ECR login credentials on behalf of the pod's assumed IAM role, making imagePullSecrets redundant for private ECR access. You would only need imagePullSecrets if pulling from a different private registry (e.g., Docker Hub Private, Azure Container Registry, Google Container Registry) or if you're not using IRSA for ECR.

Q3: My EKS worker nodes are in private subnets. What specific network configurations are required for ECR image pulls?

A3: For nodes in private subnets, you must configure VPC Interface Endpoints (AWS PrivateLink) for the following services in your VPC:

• com.amazonaws.<region>.ecr.api (for ECR API calls like GetAuthorizationToken)
• com.amazonaws.<region>.ecr.dkr (for Docker push/pull operations)
• com.amazonaws.<region>.s3 (ECR uses S3 as its underlying storage, so access to S3 is crucial for downloading image layers).

Additionally, ensure that the Security Groups associated with your worker nodes and these VPC Endpoints allow outbound HTTPS (port 443) traffic. Proper DNS resolution for these endpoints within your VPC is also critical.

Conclusion

Resolving ImagePullBackOff with private ECR on EKS using IRSA often boils down to a systematic check of IAM permissions, Kubernetes Service Account configuration, and network connectivity. By following this comprehensive guide, you can efficiently diagnose and rectify these issues, ensuring your containerized applications deploy smoothly and securely on AWS EKS. Adopting best practices for automation, least privilege, and robust networking will significantly reduce the occurrence of such problems in your cloud-native deployments.