Troubleshooting Kubernetes ImagePullBackOff for Private ECR Images on AWS EKS

Tech Note: Always backup your configuration files before applying any changes to production environments.

Troubleshooting Kubernetes ImagePullBackOff for Private ECR Images on AWS EKS

One of the most common and often frustrating issues encountered when deploying applications on Kubernetes, especially with private image repositories like AWS Elastic Container Registry (ECR) on Amazon Elastic Kubernetes Service (EKS), is the ImagePullBackOff error. This error signifies that Kubernetes was unable to pull the specified container image from the registry, preventing your pod from starting. This comprehensive guide will dissect the problem, provide a step-by-step troubleshooting manual, and offer best practices for prevention.

Symptom Analysis & Root Causes

Understanding the symptoms and underlying causes is crucial for efficient problem resolution.

Symptoms:

• ImagePullBackOff Pod Status: When you run kubectl get pods, your pod will be stuck in a Pending or CrashLoopBackOff state, with the status indicating ImagePullBackOff.
• ErrImagePull Event: Describing the pod with kubectl describe pod <pod-name> will show an ErrImagePull event in the events section, providing more specific details about the failure (e.g., "Failed to pull image...", "Unauthorized", "no basic auth credentials").
• Container Creation Errors: The pod's containers will fail to start, reporting errors related to image pulling.

Common Root Causes:

• Incorrect IAM Permissions: The IAM role associated with your EKS worker nodes (or the service account used by your pod via IRSA) lacks the necessary permissions to authenticate with ECR and pull images.
• ECR Repository Policy Restrictions: The specific ECR repository policy might explicitly deny access to the IAM role attempting to pull the image, or it may not grant permission to the necessary roles.
• Network Connectivity Issues: EKS worker nodes are unable to reach ECR endpoints. This can be due to:
  • Missing or misconfigured VPC Endpoints for ECR in private subnets.
  • Incorrect Security Group rules blocking outbound traffic to ECR.
  • Missing NAT Gateway or incorrect route table configuration for public internet access in private subnets.
• Incorrect Image Name or Tag: A typo in the image URI, repository name, or tag in your Kubernetes manifest.
• Kubelet Credential Helper Misconfiguration (Custom AMIs): While EKS-optimized AMIs come pre-configured with the ECR credential helper, custom AMIs might lack this setup, preventing the kubelet from automatically authenticating with ECR using IAM roles.
• Expired ECR Authorization Token: Although less common with EKS worker node roles (which auto-refresh), it can happen in manual scenarios.

Step-by-Step Resolution Guide

Follow these steps systematically to diagnose and resolve ImagePullBackOff errors for private ECR images on AWS EKS.

Step 1: Verify Kubernetes Events and Pod Status

Start by inspecting the affected pod to get initial error messages.

kubectl get pods -n <namespace> # Look for pods in ImagePullBackOff or ErrImagePull status kubectl describe pod <pod-name> -n <namespace> # Examine the "Events" section for detailed error messages

Pay close attention to messages like "Failed to pull image", "unauthorized", "no such host", or "timed out". These hints are crucial.

Step 2: Check EKS Node Group IAM Permissions

EKS worker nodes need specific IAM permissions to pull images from ECR. The instance profile attached to your worker nodes must have these permissions.

1. Identify Worker Node IAM Role: Find the IAM role attached to your EKS worker nodes. This is typically done by inspecting an EC2 instance in your node group.
# Get an instance ID from your node group aws ec2 describe-instances --filters "Name=tag:eks:cluster-name,Values=<your-cluster-name>" --query 'Reservations[].Instances[0].InstanceId' --output text # Describe the instance to find its IAM instance profile ARN aws ec2 describe-instances --instance-ids <instance-id> --query 'Reservations[].Instances[0].IamInstanceProfile.Arn' --output text # The ARN will look like: arn:aws:iam::123456789012:instance-profile/eks-node-group-role-1234ABCD # Get the role name from the instance profile ARN # (e.g., if profile is 'arn:aws:iam::.../instance-profile/my-role', the role name is 'my-role')
2. Verify ECR Pull Permissions: Ensure the identified IAM role has a policy granting ECR pull permissions. The required actions are:
   • ecr:GetAuthorizationToken
   • ecr:BatchCheckLayerAvailability
   • ecr:GetDownloadUrlForLayer
   • ecr:BatchGetImage

   A common AWS managed policy is AmazonEC2ContainerRegistryReadOnly or AmazonEKSWorkerNodePolicy (which includes ECR read access).

   # Check attached policies for the role (replace with your role name) aws iam list-attached-role-policies --role-name <your-node-iam-role-name> # Get policy details if needed aws iam get-policy-version --policy-arn <policy-arn> --version-id <version-id>

   If permissions are missing, attach the AmazonEC2ContainerRegistryReadOnly policy to your worker node's IAM role.

Step 3: Verify ECR Repository Policy

Even with correct IAM role permissions, the ECR repository itself might have a policy that denies access or is overly restrictive.

# Get the repository policy (replace with your ECR repository name) aws ecr get-repository-policy --repository-name <your-ecr-repo-name>

Ensure the policy explicitly allows ecr:BatchGetImage and ecr:GetDownloadUrlForLayer actions for the worker node IAM role (or the specific IAM role used by IRSA pods).

Example ECR Repository Policy allowing access for a specific role:

{ "Version": "2008-10-17", "Statement": [ { "Sid": "AllowEKSNodePull", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<your-node-iam-role-name>" }, "Action": [ "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage" ] } ] }

Step 4: Network Connectivity to ECR

Worker nodes must be able to reach the ECR service endpoints. ECR public endpoints are region-specific (e.g., ecr.us-east-1.amazonaws.com). If your nodes are in private subnets, you need either a NAT Gateway or VPC Endpoints for ECR.

1. VPC Endpoints: If using private subnets, ensure you have a VPC interface endpoint for ECR (for ecr.api) and VPC gateway endpoint for S3 (com.amazonaws.region.s3) as ECR leverages S3 for image layers.
2. Security Groups:
   • Worker Node Security Group: Must allow outbound HTTPS (port 443) traffic to ECR. If using VPC Endpoints, ensure it allows outbound to the Security Groups of the ECR VPC Endpoints.
   • VPC Endpoint Security Group (if used): Must allow inbound HTTPS (port 443) from the worker node Security Group.
3. Test Connectivity from a Worker Node: SSH into a worker node and attempt to authenticate with ECR.
# First, get an ECR authorization token # Replace <AWS_ACCOUNT_ID> and <AWS_REGION> aws ecr get-login-password --region <AWS_REGION> | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com # Then, try to pull the problematic image docker pull <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<your-ecr-repo-name>:<tag>

If docker login or docker pull fails on the worker node, you have a direct access issue. Check IAM, network routes, and security groups.

Step 5: Image Name and Tag Verification

A simple but often overlooked cause is an incorrect image URI or tag in your Deployment or Pod manifest.

# Example of a correct ECR image URI in a manifest apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 1 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-container image: <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<your-ecr-repo-name>:<tag> ports: - containerPort: 80

Ensure the image field exactly matches the ECR image URI, including account ID, region, repository name, and tag. Verify the tag actually exists in ECR.

Step 6: Kubelet Credential Helper (for Custom AMIs or Deeper Dive)

EKS-optimized AMIs automatically configure the kubelet to use aws-iam-authenticator and the ECR credential helper. This means the kubelet on the worker node uses the node's IAM role to get temporary ECR credentials.

If you are using a custom AMI or seeing issues with credential refresh, verify the kubelet configuration:

1. SSH into a worker node.
2. Check the Docker daemon configuration for a credential helper:
cat /etc/docker/daemon.json # Look for something like: # { # "credsStore": "ecr-login" # }
3. Ensure the ecr-login binary (from amazon-ecr-credential-helper) is installed and accessible in the system's PATH.
which docker-credential-ecr-login

If using a custom AMI and these are missing, you'll need to install and configure them. For most EKS setups with EKS-optimized AMIs, this step is usually not the root cause unless the AMI has been heavily modified.

Step 7: For Pods using Service Accounts (IRSA - IAM Roles for Service Accounts)

If your pod is configured to use a specific Kubernetes Service Account that is backed by an IAM Role (IRSA) for fine-grained permissions, then that IAM role, not the node's IAM role, is responsible for ECR access.

1. Identify Pod's Service Account: Check your pod's manifest for serviceAccountName.
2. Verify IAM Role Association: Ensure the Service Account is correctly associated with an IAM role. This usually involves an annotation on the service account like eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<your-irsa-role>.
3. Check IRSA IAM Role Permissions: The IAM role associated with the Service Account must have the same ECR pull permissions as outlined in Step 2.
   # Get the role name from the service account's ARN # Check attached policies for this specific IRSA role. aws iam list-attached-role-policies --role-name <your-irsa-role-name>

   If using IRSA, the worker node's IAM role still needs sts:AssumeRoleWithWebIdentity permissions for the OIDC provider associated with your EKS cluster to enable the pod to assume the IRSA role.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the occurrence of ImagePullBackOff errors.

• Least Privilege IAM Policies: Grant only the necessary ECR pull permissions to your worker node IAM roles or IRSA roles. Avoid overly broad permissions.
• Utilize IRSA for Fine-Grained Access: For applications requiring access to specific ECR repositories, use IAM Roles for Service Accounts (IRSA) to grant specific pods access without extending permissions to all nodes.
• Implement VPC Endpoints for ECR: For private subnets, use VPC Interface Endpoints for ECR API and Gateway Endpoints for S3 to ensure secure and efficient image pulling without relying on NAT Gateways or internet access.
• Consistent Image Tagging Strategy: Use immutable tags (e.g., Git SHA or build number) rather than mutable tags like latest. This ensures reproducibility and prevents unexpected image changes.
• Keep EKS Worker Nodes Up-to-Date: Regularly update your EKS worker node AMIs to benefit from the latest security patches, bug fixes, and ECR credential helper configurations.
• Automate Image Builds and Pushes: Use CI/CD pipelines to automate building and pushing images to ECR, ensuring consistent image naming and tagging.
• Proactive Monitoring: Integrate CloudWatch or other monitoring solutions to alert on ImagePullBackOff events or ECR authentication failures.

Frequently Asked Questions (FAQs)

Q1: Why do I get ImagePullBackOff instead of just ErrImagePull?

ErrImagePull is the initial error indicating that Kubernetes failed to pull the image. ImagePullBackOff is a status that Kubernetes assigns to a pod after multiple consecutive ErrImagePull failures. It signifies that Kubernetes is backing off (waiting) before attempting another pull, to prevent overwhelming the image registry or resource exhaustion.

Q2: Do I need to create a Kubernetes ImagePullSecret for ECR images on EKS?

In most standard EKS setups using EKS-optimized AMIs, you do not need to manually create an ImagePullSecret for ECR. The EKS worker nodes are configured with an ECR credential helper that automatically uses the node's IAM role (or the pod's IRSA role) to obtain temporary ECR authentication tokens, making the process seamless. ImagePullSecrets are typically used for other private registries or custom configurations.

Q3: How can I quickly test if an EKS worker node has ECR access?

The most direct way is to SSH into an affected worker node and manually try to log in to ECR and pull an image using the docker CLI:

# 1. Get ECR login password # Replace <AWS_REGION> and <AWS_ACCOUNT_ID> aws ecr get-login-password --region <AWS_REGION> | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com # 2. Attempt to pull the image # Replace with your full ECR image URI docker pull <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<your-ecr-repo-name>:<tag>

Success indicates the node has proper IAM permissions and network connectivity. Failure points directly to issues with IAM, network, or the ECR credential helper setup on that specific node.