Debugging EKS Pod ImagePullBackOff Errors from AWS ECR with IAM Roles for Service Accounts

Tech Note: Always backup your configuration files before applying any changes to production environments.

Debugging EKS Pod ImagePullBackOff Errors from AWS ECR with IAM Roles for Service Accounts (IRSA)

Encountering an ImagePullBackOff error in your Amazon Elastic Kubernetes Service (EKS) cluster can be a frustrating roadblock, especially when your container images reside in AWS Elastic Container Registry (ECR). This issue typically indicates that your Kubernetes pod is unable to pull the required container image. When using IAM Roles for Service Accounts (IRSA) to grant permissions, the problem often boils down to misconfigured IAM policies, trust relationships, or Kubernetes service account annotations. This comprehensive guide and troubleshooting manual will equip you with the knowledge and steps to diagnose and resolve these common EKS image pull failures.

Symptom Analysis & Root Causes

The primary symptom of this issue is a pod stuck in a Pending or ContainerCreating state, with events indicating ImagePullBackOff or ErrImagePull. To confirm, you would typically see output similar to this:

kubectl get pods NAME READY STATUS RESTARTS AGE my-app-pod-xyz 0/1 ImagePullBackOff 0 5m kubectl describe pod my-app-pod-xyz ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m default-scheduler Successfully assigned default/my-app-pod-xyz to ip-10-0-1-123.ec2.internal Normal Pulling 4m (x2 over 5m) kubelet Pulling image "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest" Warning Failed 4m (x2 over 5m) kubelet Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://123456789012.dkr.ecr.us-east-1.amazonaws.com/v2/": no basic auth credentials Warning Failed 4m (x2 over 5m) kubelet Error: ImagePullBackOff Normal BackOff 2m (x3 over 5m) kubelet Back-off pulling image "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest" ...

Common Root Causes:

• Incorrect IAM Policy for ECR Access: The IAM Role associated with the service account lacks the necessary permissions (e.g., ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, ecr:BatchGetImage) to pull images from ECR.
• Misconfigured IAM Role for Service Account (IRSA):
  • Missing or Incorrect Service Account Annotation: The Kubernetes Service Account (SA) in the pod's namespace does not have the eks.amazonaws.com/role-arn annotation pointing to the correct IAM Role.
  • IAM Role Trust Policy Issues: The IAM Role's Trust Policy does not correctly allow the EKS OIDC provider to assume the role, or the OIDC provider URL is incorrect.
  • Incorrect IAM Role ARN: The ARN specified in the Service Account annotation is incorrect or refers to a non-existent role.
• EKS OIDC Provider Not Set Up Correctly: The OpenID Connect (OIDC) provider for your EKS cluster might not be enabled or correctly configured, which is essential for IRSA.
• ECR Repository Policy Restrictions: The ECR repository itself might have a policy that explicitly denies access to the IAM Role or the AWS account attempting the pull.
• Network Connectivity Issues:
  • VPC Endpoints (Interface ECR): If using private networks, the ECR VPC endpoint might be misconfigured, missing, or its security groups/network ACLs might be blocking traffic from EKS worker nodes.
  • Internet Gateway/NAT Gateway: If not using VPC endpoints, worker nodes might lack outbound internet access to reach public ECR endpoints.
• Image Name or Tag Errors: A simple typo in the image name or tag specified in the pod manifest.

Step-by-Step Resolution Guide

Follow these steps methodically to pinpoint and resolve the ImagePullBackOff error.

Step 1: Verify Pod Status and Events

Start by getting detailed information about the failing pod.

kubectl get pod <pod-name> -n <namespace> kubectl describe pod <pod-name> -n <namespace>

Look for messages in the Events section that explicitly mention authentication failures, network issues, or "no basic auth credentials" related to ECR. Note the image name/tag it's trying to pull.

Step 2: Check Kubernetes Service Account and Pod Configuration

Identify the Service Account used by your pod. If none is specified, it defaults to default in its namespace.

kubectl get serviceaccount <service-account-name> -n <namespace> -o yaml

Ensure the Service Account has the correct IRSA annotation:

apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-ecr-pull-role name: my-app-service-account namespace: default

Verify the pod manifest correctly references this service account:

apiVersion: v1 kind: Pod metadata: name: my-app-pod namespace: default spec: serviceAccountName: my-app-service-account containers: - name: my-app image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest ports: - containerPort: 80

Double-check the image name and tag for typos.

Step 3: Validate AWS IAM Role and Trust Policy

Access the AWS IAM console and locate the IAM Role specified in the Service Account annotation (e.g., my-ecr-pull-role).

• Permissions Policy: Ensure the role has a policy attached that grants ECR pull permissions. A common policy would look like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:DescribeImages" ], "Resource": "*" } ] }

• Trust Policy: Navigate to the "Trust relationships" tab for the IAM Role. It must allow the EKS OIDC provider to assume the role. The policy should resemble:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EXAMPLED539D4633E53BB2B3F05C384" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.REGION.amazonaws.com/id/EXAMPLED539D4633E53BB2B3F05C384:sub": "system:serviceaccount:default:my-app-service-account" } } } ] }

Important:

• Replace 123456789012 with your AWS Account ID.
• Replace REGION with your AWS region (e.g., us-east-1).
• Replace EXAMPLED539D4633E53BB2B3F05C384 with your EKS cluster's OIDC provider ID. You can find this by running:

aws eks describe-cluster --name <your-cluster-name> --query "cluster.identity.oidc.issuer" --output text

The Condition block ensures that only the specified service account can assume the role. If you want to allow multiple SAs, consider using StringLike or adding multiple conditions.
Also, ensure that your EKS cluster has an OIDC Identity Provider configured. You can check this in the EKS console or by using the AWS CLI:

aws iam list-open-id-connect-providers

If the provider is missing, you'll need to create it for your EKS cluster. Tools like eksctl simplify this: eksctl utils associate-iam-oidc-provider --cluster <your-cluster-name> --approve

Step 4: Check ECR Repository Policy

If your IAM Role and Trust Policy seem correct, check the ECR repository policy. Go to the ECR console, select your repository, and view its "Permissions" tab. Ensure there isn't an explicit deny for the role or account, or that your role is explicitly allowed if the policy is restrictive.

Step 5: Validate Network Connectivity (VPC Endpoints/Security Groups)

This is critical if your EKS worker nodes are in private subnets without direct internet access.

• VPC Endpoint for ECR: Verify you have a VPC interface endpoint for ECR (e.g., com.amazonaws.REGION.ecr.dkr and com.amazonaws.REGION.s3 or com.amazonaws.REGION.ecr.api).
• Security Groups: The security group attached to the ECR VPC endpoint must allow inbound HTTPS (port 443) traffic from the security groups of your EKS worker nodes. Conversely, the worker node security groups must allow outbound HTTPS to the ECR endpoint security group.
• Network ACLs: Ensure network ACLs associated with the subnets allow inbound/outbound HTTPS traffic.
• Worker Node Internet Access: If not using VPC endpoints, ensure worker nodes have a route to an Internet Gateway (public subnets) or a NAT Gateway (private subnets) to reach the public ECR endpoints.

Step 6: Test ECR Pull from Worker Node (Advanced Debugging)

For deep dives into network or credential issues, you can SSH into a worker node running the failing pod and attempt to pull the image manually. This often requires temporarily installing Docker/containerd client and AWS CLI.

# SSH into an EKS worker node ssh -i /path/to/key.pem ec2-user@<worker-node-ip> # Assuming AWS CLI is installed and configured (or you can install it) # Get a temporary ECR login password aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<your-region>.amazonaws.com # Attempt to pull the image docker pull <aws-account-id>.dkr.ecr.<your-region>.amazonaws.com/my-repo:latest

If this fails, the error message will be more explicit about the underlying issue (e.g., "no such host", "access denied").

Step 7: Apply Fixes and Retest

After identifying and correcting the misconfiguration (IAM policy, Trust Policy, Service Account annotation, ECR policy, or network settings), delete the failing pod to force Kubernetes to reschedule it with the updated configuration.

kubectl delete pod <pod-name> -n <namespace>

Monitor the pod status and events again: kubectl get pod <pod-name> -n <namespace> and kubectl describe pod <pod-name> -n <namespace>.

Best Practices for Prevention & Performance Optimization

Preventing ImagePullBackOff errors requires good architectural and operational practices:

• Automate IRSA Creation: Use tools like eksctl to create and manage your IAM Roles for Service Accounts. It automatically handles the OIDC provider association and trust policy setup.
• Least Privilege IAM Policies: Grant only the necessary ECR pull permissions to the IAM Role. Avoid using "Resource": "*" unless absolutely necessary; instead, specify target ECR repository ARNs.
• Dedicated Service Accounts: Create specific service accounts for applications requiring ECR access, rather than relying on the default SA.
• Leverage ECR Repository Policies: Use ECR repository policies to allow specific accounts or roles to pull images, especially in multi-account environments. This adds an extra layer of security.
• VPC Endpoints for Private Connectivity: Always configure ECR interface VPC endpoints (ecr.api and ecr.dkr) for EKS clusters in private subnets. This enhances security, reduces data transfer costs, and provides predictable network performance.
• Monitor EKS and ECR Logs: Integrate EKS control plane logs and CloudTrail logs (for ECR API calls) into CloudWatch or a centralized logging solution. This helps in quickly identifying authentication or authorization failures.
• Image Hygiene and Versioning: Maintain clear image naming conventions and use specific, immutable tags (e.g., Git SHA, build number) instead of latest. This ensures consistency and reproducibility.
• Regular Audits: Periodically audit your IAM policies, service account configurations, and ECR repository settings to ensure they align with security and operational best practices.

Frequently Asked Questions (FAQs)

Q1: Why are my pods stuck in ContainerCreating after ImagePullBackOff?

A pod transitioning from ImagePullBackOff to ContainerCreating indicates that the image pull succeeded on a subsequent retry. However, it then failed at the container creation stage. This could be due to:

• Application Errors: The container tried to start but immediately exited due to an application-level error (e.g., incorrect entrypoint, missing dependencies, bad configuration). Check kubectl logs <pod-name> -n <namespace> and kubectl describe pod <pod-name> -n <namespace> events for clues about container startup issues.
• Resource Limits: Insufficient CPU or memory resources defined in the pod spec.
• Probes Failing: Liveness or readiness probes failing, causing Kubernetes to restart the container.

Focus your debugging efforts on the container logs and events after a successful image pull.

Q2: How do I ensure my EKS cluster can pull images from private ECR repositories in another AWS account?

Pulling from another account's private ECR requires two main configurations:

1. IAM Role in EKS Account: The IAM Role attached to your EKS Service Account (in the EKS cluster's account) needs permissions to pull images from the other account's ECR. This involves:
   • Adding ECR pull actions (ecr:GetAuthorizationToken, etc.) to the EKS account's IAM Role.
   • Specifying the resource ARNs for the cross-account ECR repositories in the policy.
2. ECR Repository Policy in Target Account: The ECR repository (in the other AWS account) must have a policy that explicitly allows the IAM Role from your EKS account to perform the necessary ECR actions. This policy will typically use a Principal clause with the ARN of the IAM Role from the EKS account.

Both sides must grant permission for the cross-account pull to succeed.

Q3: Can I use imagePullSecrets with IRSA? When is it necessary?

While IRSA is the recommended and most secure way to authenticate with ECR for EKS pods, imagePullSecrets can still be used, though less common for ECR.
When it's necessary/useful:

• Non-ECR Registries: When pulling images from third-party private registries (Docker Hub, Quay.io, Azure Container Registry, Google Container Registry, etc.). IRSA is specific to AWS IAM.
• Legacy Applications: For applications not yet migrated to IRSA or in environments where IRSA isn't fully adopted.
• Fallback/Specific Scenarios: In complex multi-registry scenarios where you might need to combine authentication methods.

For ECR specifically, IRSA is preferred as it avoids managing long-lived credentials and provides fine-grained, pod-level permissions. If you find yourself needing imagePullSecrets for ECR, double-check if IRSA could be configured to achieve the same securely.