Resolving Kubernetes ImagePullBackOff on AWS EKS with Private ECR Repositories

Tech Note: Always backup your configuration files before applying any changes to production environments.

Resolving Kubernetes ImagePullBackOff on AWS EKS with Private ECR Repositories

The ImagePullBackOff error is a common frustration for developers and DevOps engineers working with Kubernetes. When this error occurs, your pods are stuck trying to pull a container image, indicating a problem with accessing the image registry. This guide focuses specifically on scenarios involving AWS Elastic Kubernetes Service (EKS) and private Amazon Elastic Container Registry (ECR) repositories, a frequent setup in cloud-native environments. We'll delve into the root causes and provide a comprehensive, step-by-step troubleshooting manual to get your deployments back on track.

Symptom Analysis & Root Causes

The ImagePullBackOff status in Kubernetes signifies that a pod has failed to pull a container image from a specified registry. The "BackOff" part indicates that Kubernetes is retrying the pull operation, but to no avail. Understanding the underlying reasons is crucial for an effective resolution, especially when dealing with private registries like AWS ECR in an EKS environment.

Common Symptoms:

• Pod Status: Pods remaining in Pending or ImagePullBackOff status.
• Events: Running kubectl describe pod <pod-name> shows events like Failed to pull image "ecr-repo/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://<aws_account_id>.dkr.ecr.<region>.amazonaws.com/v2/<repo_name>/manifests/<tag>": no basic auth credentials or no such host.
• Kubelet Logs: Logs on the worker node show similar image pull failures.

Primary Root Causes for EKS/Private ECR:

• IAM Permissions Insufficiency: This is the most frequent culprit. The IAM role associated with your EKS worker nodes (or the service account if using IRSA) lacks the necessary permissions to authenticate with ECR and pull images.
  • Required Permissions: ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:BatchGetImage.
  • Scope: Permissions can be missing from the EKS NodeInstanceRole, or if using IAM Roles for Service Accounts (IRSA), the specific IAM Role attached to the Kubernetes Service Account.
• ECR Repository Policy: Even if your IAM role has the necessary permissions, the ECR repository itself might have a resource policy that explicitly denies access or doesn't permit access from your specific IAM role/account. This is common in cross-account scenarios.
• Incorrect Image Name or Tag: A simple typo in the image name, repository path, or tag in your Kubernetes deployment manifest will lead to this error. The image might not exist at the specified location.
• Network Connectivity Issues:
  • Security Groups: The security group attached to your EKS worker nodes might not allow outbound HTTPS (port 443) traffic to ECR endpoints.
  • VPC Endpoints (Private ECR Access): If ECR is configured for private access via VPC Endpoints, ensure the endpoint is correctly configured, associated with the right subnets, and its security groups allow traffic from theKS worker nodes. Also, DNS resolution must be correctly configured for the ECR endpoint.
  • NAT Gateway/Internet Gateway: If not using VPC endpoints, worker nodes need a route to an Internet Gateway or NAT Gateway for outbound ECR access.
• Kubernetes ImagePullSecrets: While less common with EKS's native IAM integration, if you're explicitly using imagePullSecrets, they might be incorrect, expired, or missing.

Step-by-Step Resolution Guide

Follow these steps to systematically diagnose and resolve ImagePullBackOff errors on AWS EKS with private ECR repositories.

Prerequisites:

• AWS CLI configured with appropriate permissions.
• kubectl configured to interact with your EKS cluster.
• jq for parsing JSON output (optional but highly recommended).

Step 1: Verify Image Name and Tag in Kubernetes Manifest

Start with the basics. A simple typo can save hours of complex troubleshooting.

Action: Check your deployment, pod, or daemonset manifest for the correct ECR image URI and tag.

kubectl get pod <pod-name> -o yaml | grep image:

Ensure the image URI matches the exact ECR repository path (e.g., <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag>).

# Example of a correct image URI image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app/backend:v1.0.0

Step 2: Examine Pod Events and Status

kubectl describe provides valuable insights into why a pod failed.

Action: Describe the failing pod and look for events related to image pulling.

kubectl describe pod <pod-name> -n <namespace>

Look for messages under the "Events" section, particularly those with "Failed" in their description. These often indicate permission denied, network issues, or image not found.

Step 3: Verify IAM Permissions for ECR Access

This is where most ECR image pull issues arise. EKS worker nodes (or specific service accounts) need explicit permissions to interact with ECR.

Sub-step 3.1: Check EKS Worker Node IAM Role Permissions

EKS worker nodes assume an IAM role (e.g., eksctl-<cluster-name>-nodegroup-<nodegroup>-NodeInstanceRole or a custom role). This role needs ECR access.

Action:

1. Find the Worker Node IAM Role:
   # Get an EKS worker node name NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') # Get the EC2 instance ID for that node INSTANCE_ID=$(kubectl get node $NODE_NAME -o jsonpath='{.spec.providerID}' | cut -d/ -f2) # Get the IAM instance profile ARN INSTANCE_PROFILE_ARN=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn' --output text) # Extract the IAM role name from the instance profile ARN NODE_IAM_ROLE_NAME=$(echo $INSTANCE_PROFILE_ARN | cut -d'/' -f2) echo "Worker Node IAM Role: $NODE_IAM_ROLE_NAME"
2. Verify ECR Permissions on the Role: Check if the role has policies attached that grant the following permissions:
   • ecr:GetAuthorizationToken
   • ecr:BatchCheckLayerAvailability
   • ecr:GetDownloadUrlForLayer
   • ecr:BatchGetImage

   The AWS managed policy AmazonEKSWorkerNodePolicy (for EKS nodes) and AmazonEC2ContainerRegistryReadOnly or a custom policy often provide these.

   # List attached policies for the role aws iam list-attached-role-policies --role-name $NODE_IAM_ROLE_NAME # You may need to inspect each policy's content to ensure ECR permissions # Example for an inline policy: # aws iam get-role-policy --role-name $NODE_IAM_ROLE_NAME --policy-name # Example for a managed policy: # aws iam get-policy-version --policy-arn --version-id

   Resolution: If missing, attach AmazonEC2ContainerRegistryReadOnly or a custom policy with the required permissions to the worker node IAM role.

Sub-step 3.2: Check IAM Roles for Service Accounts (IRSA) Permissions

If your pods are configured to use IRSA for ECR access, the permissions must be on the IAM role associated with the Kubernetes Service Account.

Action:

1. Identify the Service Account and its IAM Role:
   # Get the service account associated with your pod kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}' # Describe the service account to find the IAM Role ARN SERVICE_ACCOUNT_NAME=$(kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.serviceAccountName}') kubectl describe serviceaccount $SERVICE_ACCOUNT_NAME -n <namespace> | grep "eks.amazonaws.com/role-arn" # Extract the role name IAM_ROLE_ARN=$(kubectl describe serviceaccount $SERVICE_ACCOUNT_NAME -n <namespace> | grep "eks.amazonaws.com/role-arn" | awk '{print $2}') IRSA_IAM_ROLE_NAME=$(echo $IAM_ROLE_ARN | cut -d'/' -f2) echo "IRSA IAM Role: $IRSA_IAM_ROLE_NAME"
2. Verify ECR Permissions on the IRSA Role: Use the same method as for worker node roles (Sub-step 3.1) to check if the IRSA_IAM_ROLE_NAME has the necessary ECR read permissions.
3. Verify OIDC Provider: Ensure your EKS cluster has an OIDC Identity Provider configured and that the IAM Trust Policy for the IRSA role allows the OIDC provider to assume the role.
   # Get OIDC provider for your EKS cluster aws eks describe-cluster --name <your-cluster-name> --query "cluster.identity.oidc.issuer" --output text # Check IAM role's trust policy for the OIDC provider aws iam get-role --role-name $IRSA_IAM_ROLE_NAME --query 'Role.AssumeRolePolicyDocument'

   The trust policy should contain something similar to:

   { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<aws_account_id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<oidc_id>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.<region>.amazonaws.com/id/<oidc_id>:sub": "system:serviceaccount:<namespace>:<service_account_name>" } } }

Step 4: Check ECR Repository Policy

Even with correct IAM permissions, an ECR repository policy can restrict access.

Action: Retrieve the ECR repository policy and ensure it allows your EKS worker node role or IRSA role to pull images.

REPOSITORY_NAME="<your-repository-name>" # e.g., my-app/backend aws ecr get-repository-policy --repository-name $REPOSITORY_NAME --query "policyText" --output text | jq '.'

Look for explicit "Deny" statements that might affect your role or "Allow" statements that don't include your role's ARN. For cross-account ECR pulls, the repository policy must explicitly grant permissions to the IAM role in the other account.

# Example ECR Repository Policy snippet allowing access from an EKS worker role { "Sid": "AllowEKSNodeAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<aws_account_id>:role/<your_eks_node_iam_role_name>" }, "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability" ] }

Step 5: Diagnose Network Connectivity

Network issues can prevent worker nodes from reaching ECR.

Sub-step 5.1: Check Worker Node Security Groups

Action: Ensure the security group attached to your EKS worker nodes allows outbound HTTPS (port 443) traffic to ECR.

• Go to the EC2 console, find an EKS worker instance, and check its associated security groups.
• In the outbound rules, there should be a rule allowing HTTPS (Port 443) to 0.0.0.0/0 (for public ECR access) or to the VPC Endpoint Security Group/CIDR (for private ECR access).

Sub-step 5.2: Verify VPC Endpoints (if used)

If your EKS cluster is in a private subnet and accesses ECR privately via a VPC Endpoint (interface endpoint for ECR API and ECR DKR), ensure it's correctly configured.

Action:

1. Check for ECR VPC Endpoints:
   aws ec2 describe-vpc-endpoints --filters "Name=service-name,Values=com.amazonaws.<region>.ecr.dkr" "Name=service-name,Values=com.amazonaws.<region>.ecr.api" --query "VpcEndpoints[*].{VpcEndpointId:VpcEndpointId,ServiceName:ServiceName,State:State,SubnetIds:SubnetIds}"

   Ensure endpoints for ecr.api and ecr.dkr exist and are in the available state.

2. Subnet Association: Verify the VPC Endpoints are associated with the same subnets where your EKS worker nodes reside.
3. Security Group for Endpoint: The security group attached to the VPC Endpoint must allow inbound HTTPS (port 443) from your EKS worker node security groups.
4. Route Tables: Ensure the route tables for your worker node subnets direct traffic to the VPC Endpoints for ECR.
5. DNS Resolution: Private DNS for the VPC Endpoint should be enabled, or you must configure DNS resolution correctly for ECR service names to resolve to private IPs.

Step 6: Manual Docker Login from Worker Node (for Debugging)

This step helps isolate if the issue is with the worker node's ability to authenticate and pull from ECR, independent of Kubernetes.

Action: SSH into an EKS worker node that's failing to pull images and attempt a manual Docker login and image pull.

# On the EKS worker node: # 1. Get ECR login credentials aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com # 2. Attempt to pull the problematic image docker pull <aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag>

If the docker login or docker pull fails on the worker node, it points to a problem with the worker node's IAM role (if not using IRSA), network connectivity, or ECR repository policy. If it succeeds, the issue might be more subtle within Kubernetes' image pull mechanism, suggesting a misconfigured Service Account or incorrect imagePullSecrets (if used).

Best Practices for Prevention & Performance Optimization

Adopting these best practices can significantly reduce the occurrence of ImagePullBackOff errors and enhance the security and efficiency of your EKS deployments.

• Leverage IAM Roles for Service Accounts (IRSA):
  • Benefit: Granular permissions. Instead of giving broad ECR access to all worker nodes, IRSA allows you to grant specific ECR read permissions only to the pods that need them. This enhances your security posture significantly.
  • Implementation: Annotate your Kubernetes Service Account with the ARN of an IAM role that has the necessary ECR read permissions.
• Implement Strict ECR Repository Policies:
  • Benefit: Control who can push/pull from specific repositories, especially important for sensitive images or cross-account access.
  • Implementation: Define resource-based policies on your ECR repositories to explicitly allow only trusted IAM roles or accounts.
• Use VPC Endpoints for ECR:
  • Benefit: Pull images over a private network connection, improving security, reducing data transfer costs (for cross-region/cross-AZ traffic), and providing consistent performance.
  • Implementation: Create interface VPC endpoints for com.amazonaws.<region>.ecr.api and com.amazonaws.<region>.ecr.dkr in your VPC.
• Automate Image Scanning:
  • Benefit: Detect vulnerabilities in your container images before deployment.
  • Implementation: Enable ECR's built-in image scanning or integrate with third-party tools in your CI/CD pipeline.
• Consistent Image Tagging Strategy:
  • Benefit: Avoid confusion and accidental deployment of incorrect image versions.
  • Implementation: Use semantic versioning (e.g., v1.2.3) or Git commit SHAs for tags. Avoid :latest in production as it can lead to non-reproducible deployments.
• Monitor and Alert:
  • Benefit: Proactively identify and respond to image pull failures.
  • Implementation: Set up CloudWatch alarms for EKS cluster logs and ECR events. Use tools like Prometheus and Grafana to monitor Kubernetes events.

Frequently Asked Questions (FAQs)

Q1: Why am I getting ImagePullBackOff even though my EKS worker node IAM role has AmazonEC2ContainerRegistryReadOnly?

A1: While AmazonEC2ContainerRegistryReadOnly is usually sufficient for worker nodes, other factors can cause this.

• ECR Repository Policy: The ECR repository itself might have an explicit "Deny" policy for your account or role, overriding the IAM role's permissions.
• Network Connectivity: Even with correct IAM, if your worker node cannot reach the ECR endpoint (due to security groups, NACLs, missing NAT/Internet Gateway, or misconfigured VPC Endpoints), image pulls will fail.
• Incorrect Image URI: A simple typo in the image name or tag in your deployment manifest can lead to a "not found" error, often manifesting as ImagePullBackOff.
• Kubelet Caching/Stale Credentials: In rare cases, Kubelet might have stale ECR credentials. Restarting the Kubelet service on the affected node (or terminating and allowing EKS to replace the node) can sometimes resolve this, but it usually points to a deeper configuration issue.

Q2: Do I need to use imagePullSecrets in my Kubernetes manifests when using EKS with private ECR?

A2: Generally, no. One of the key advantages of running Kubernetes on AWS EKS is its deep integration with AWS IAM. EKS automatically configures Kubelet on worker nodes to assume the instance profile's IAM role (or the IAM role for a Service Account via IRSA) to obtain temporary ECR credentials. This eliminates the need to manually create and manage imagePullSecrets, which often contain base64 encoded Docker login credentials. You only typically need imagePullSecrets for non-ECR private registries or if your ECR setup is non-standard (e.g., pulling from a different AWS account without proper cross-account IAM federation).

Q3: How do I pull images from a private ECR repository in a different AWS account to my EKS cluster?

A3: This requires a combination of IAM roles and ECR repository policies:

1. Source Account (ECR Owner):
   • Modify the ECR repository policy to explicitly allow the IAM role of your EKS cluster's worker nodes (or the IRSA role) from the target AWS account to perform ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, and ecr:BatchCheckLayerAvailability actions.
2. Target Account (EKS Cluster):
   • Ensure the EKS worker node IAM role (or the specific IRSA role) has permissions to call ecr:GetAuthorizationToken in its own account to get a login token, and optionally sts:AssumeRole if the cross-account access is structured that way.
   • The Kubelet will use the GetAuthorizationToken from its *own* account, and then attempt to pull from the cross-account ECR repository. The ECR repository policy in the *source* account will then determine if that token (associated with the EKS worker's identity) is authorized to pull the image.
3. Image URI: The image URI in your Kubernetes manifest must include the source account ID and region (e.g., <source_account_id>.dkr.ecr.<source_region>.amazonaws.com/<repo_name>:<tag>).