Fixing EKS Pod ImagePullBackOff from Private ECR with IAM Roles for Service Accounts

Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing EKS Pod ImagePullBackOff from Private ECR with IAM Roles for Service Accounts (IRSA)

Running containerized applications on Amazon Elastic Kubernetes Service (EKS) often involves pulling images from private repositories like Amazon Elastic Container Registry (ECR). A common challenge arises when EKS pods fail to pull these images, resulting in an ImagePullBackOff status. This issue frequently points to authentication or authorization problems, especially when leveraging IAM Roles for Service Accounts (IRSA) to grant pods fine-grained permissions to ECR. This guide provides a comprehensive breakdown of symptoms, root causes, and a step-by-step troubleshooting manual to resolve ImagePullBackOff issues stemming from ECR access with IRSA.

Symptom Analysis & Root Causes

Understanding the symptoms and underlying causes is the first step towards a swift resolution. An ImagePullBackOff status means that the Kubelet on the EKS worker node repeatedly tried to pull a container image but failed.

Common Symptoms:

• Pods remain in Pending or CrashLoopBackOff state, with the primary reason often being ImagePullBackOff.
• kubectl describe pod <pod-name> output shows events like:
  • Failed to pull image "xxxxxxxx.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>": rpc error: code = Unknown desc = Error response from daemon: Get "https://xxxxxxxx.dkr.ecr.<region>.amazonaws.com/v2/": no basic auth credentials
  • Failed to pull image "xxxxxxxx.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>": rpc error: code = Unknown desc = failed to resolve reference "xxxxxxxx.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>": failed to do request: Head "https://xxxxxxxx.dkr.ecr.<region>.amazonaws.com/v2/<repo>/manifests/<tag>": dial tcp: lookup xxxxxxxx.dkr.ecr.<region>.amazonaws.com: no such host
  • Back-off pulling image "xxxxxxxx.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>"
• No logs are emitted from the application container itself, as it never starts.

Root Causes for ImagePullBackOff with IRSA and ECR:

• Incorrect IAM Policy for ECR Read Access: The IAM role associated with the Kubernetes Service Account lacks the necessary permissions (e.g., ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, ecr:BatchCheckLayerAvailability) to pull images from ECR.
• Misconfigured IAM Role Trust Policy: The IAM role's trust policy does not correctly establish trust with the EKS cluster's OpenID Connect (OIDC) provider, preventing the Kubernetes Service Account from assuming the role. This typically involves an incorrect OIDC issuer URL or a malformed condition for the OIDC provider.
• Missing or Incorrect Service Account Annotation: The Kubernetes Service Account used by the pod is not annotated with eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<iam-role-name>, or the ARN points to a non-existent or incorrect IAM role.
• EKS Cluster OIDC Provider Not Enabled/Associated: The EKS cluster must have an OIDC identity provider associated with it for IRSA to function. If this is missing, IRSA will fail.
• Network Connectivity Issues:
  • Security Groups/NACLs: EKS worker nodes lack outbound access to ECR endpoints (public or VPC endpoint) on port 443.
  • VPC Endpoints (if used): Misconfigured ECR VPC endpoint policies, incorrect DNS resolution, or missing routes to the endpoint.
  • Proxy Configuration: If using an HTTP/S proxy, the Kubelet or Docker daemon might not be correctly configured to use it for ECR access.
• Incorrect Image Name or Tag: A simple typo in the image path or an incorrect tag (e.g., trying to pull latest when it doesn't exist or is outdated) can lead to this error.

Step-by-Step Resolution Guide

Follow these steps to diagnose and resolve ImagePullBackOff issues related to ECR and IRSA.

Prerequisites:

• AWS CLI installed and configured with appropriate permissions.
• kubectl installed and configured to connect to your EKS cluster.
• eksctl (recommended for easier management of EKS clusters and IRSA).

Step 1: Verify EKS Cluster OIDC Provider

IRSA relies on an OIDC provider being associated with your EKS cluster.

aws eks describe-cluster --name <your-cluster-name> --query "cluster.identity.oidc.issuer" --output text

You should see an output like https://oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>. Make a note of this URL and the <OIDC_ID>.

Next, confirm this provider exists in IAM:

aws iam list-open-id-connect-providers | grep "<OIDC_ID>"

If the OIDC provider is missing, you'll need to create it. For eksctl users, this is often handled automatically when creating the cluster or via eksctl utils associate-iam-oidc-provider.

Step 2: Inspect Pod Status and Events

Get details on the problematic pod to identify the exact error message.

kubectl get pods -n <namespace> kubectl describe pod <pod-name> -n <namespace>

Look for Events: section. It will show the ImagePullBackOff error and often a more specific reason for the failure.

Step 3: Check Kubernetes Service Account Configuration

Confirm that your pod is using a Service Account, and that Service Account is correctly annotated for IRSA.

kubectl get serviceaccount <service-account-name> -n <namespace> -o yaml

Look for the annotations field. It should contain an entry like:

annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<your-iam-role-name>

If this annotation is missing or incorrect, you need to add/correct it on the Service Account YAML and reapply.

Step 4: Verify IAM Role Trust Policy and Permissions

Using the IAM Role ARN from Step 3, check its trust policy and attached permissions. Extract the role name from the ARN (e.g., from arn:aws:iam::123456789012:role/my-ecr-role, the role name is my-ecr-role).

a. Check Trust Policy:

The trust policy must allow the OIDC provider to assume the role.

aws iam get-role --role-name <your-iam-role-name> --query 'Role.AssumeRolePolicyDocument'

The output (decoded JSON) should look similar to this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:<namespace>:<service-account-name>", "oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:aud": "sts.amazonaws.com" } } } ] }

Key checks:

• Federated ARN matches your EKS OIDC provider (from Step 1).
• Condition.StringEquals has the correct oidc.eks.../id/<OIDC_ID>:sub value, matching your service account's namespace and name.
• oidc.eks.../id/<OIDC_ID>:aud is sts.amazonaws.com.

b. Check Permissions Policy:

Ensure the IAM role has permissions to pull images from ECR.

aws iam list-attached-role-policies --role-name <your-iam-role-name>

Review the attached policies. For ECR pull access, the policy should include actions like:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:GetAuthorizationToken" ], "Resource": [ "arn:aws:ecr:<region>:<AWS_ACCOUNT_ID>:repository/<your-ecr-repository-name>", "arn:aws:ecr:<region>:<AWS_ACCOUNT_ID>:repository/*" // If you need access to all repos ] } ] }

Note: While ecr:GetAuthorizationToken is often included, Kubelet on EKS handles ECR authentication differently via IRSA, using direct API calls. However, it doesn't hurt to include it. The critical permissions are GetDownloadUrlForLayer, BatchGetImage, and BatchCheckLayerAvailability.

Step 5: Test ECR Access from within a Pod

To isolate if the issue is with the IAM permissions or something else (like networking), create a temporary debug pod using the same service account and try to authenticate with ECR.

kubectl run -it --rm debug-pod --image=amazon/aws-cli --serviceaccount=<service-account-name> -n <namespace> -- bash

Once inside the pod, try to get an ECR login password:

aws ecr get-login-password --region <region>

If this command succeeds and returns a password, your IRSA setup for ECR authentication is likely correct. If it fails with an access denied error, review the IAM role policies (Step 4). If it hangs or shows a network error, proceed to Step 6.

Step 6: Network Connectivity Checks (If previous steps passed or showed network errors)

Even with correct IAM permissions, network restrictions can prevent EKS nodes from reaching ECR.

• Security Groups: Ensure the Security Group attached to your EKS worker nodes allows outbound HTTPS (port 443) traffic to ECR. If using ECR VPC Endpoints, ensure outbound to the VPC Endpoint's Security Group is allowed.
• NACLs: Review Network Access Control Lists (NACLs) for the subnets where your EKS nodes reside. They should allow ephemeral ports for return traffic.
• Route Tables: Verify that the subnets have routes to the internet (via Internet Gateway or NAT Gateway) or to the ECR VPC Endpoint.
• VPC Endpoints (if applicable):
  • Is the VPC Endpoint for ECR configured correctly in the same VPC as your EKS cluster?
  • Is the Endpoint Policy allowing access from your EKS cluster's network?
  • Is Private DNS enabled on the endpoint, and is DNS resolution working correctly for *.dkr.ecr.<region>.amazonaws.com?

Step 7: Double-Check Image Name and Tag

Confirm that the image name and tag specified in your pod or deployment manifest are absolutely correct and exist in your ECR repository.

aws ecr describe-images --repository-name <your-ecr-repository-name> --region <region>

Verify that the tag you're trying to pull is listed in the output.

Step 8: Reapply Pod/Deployment

After making any changes to IAM roles, service accounts, or pod manifests, ensure your changes are applied. For deployments, initiate a rollout restart.

kubectl rollout restart deployment <your-deployment-name> -n <namespace> # Or, if it's a static pod or other resource: kubectl delete pod <pod-name> -n <namespace> # Kubernetes will recreate it.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the occurrence of ImagePullBackOff errors and improve overall EKS reliability.

• Principle of Least Privilege: Grant IAM roles only the exact permissions needed (e.g., read-only access to specific ECR repositories). Avoid blanket ecr:* or Resource: "*" if not strictly necessary.
• Automate IRSA Setup: Use tools like eksctl to manage EKS clusters and IRSA. eksctl create iamserviceaccount simplifies role creation and trust policy configuration.
• Consistent Image Tagging: Adopt a clear and consistent image tagging strategy (e.g., semantic versioning, Git SHAs) and avoid using mutable tags like latest in production to ensure reproducibility.
• ECR Lifecycle Policies: Implement lifecycle policies in ECR to automatically clean up old or untagged images, preventing repository bloat and ensuring only relevant images are available.
• ECR Image Scanning: Enable ECR image scanning to detect vulnerabilities, ensuring the security of your container images.
• VPC Endpoints for ECR: For private subnets, configure VPC Endpoints for ECR to keep image pull traffic within your VPC, improving security and potentially performance. Ensure endpoint policies are configured correctly.
• Monitor ECR and EKS Logs: Set up CloudWatch logging for ECR events and ensure EKS control plane logs are enabled. This helps in auditing access and identifying unauthorized pull attempts or other issues.

Frequently Asked Questions (FAQs)

Q1: Why is ecr:GetAuthorizationToken often not strictly necessary for EKS pods pulling from ECR with IRSA?

When EKS uses IRSA for ECR, the Kubelet on the worker node obtains temporary AWS credentials by assuming the IAM role via the OIDC provider. It then uses these credentials to make direct AWS API calls to ECR (ecr:GetDownloadUrlForLayer, etc.) to retrieve image manifests and layers. It does not typically need to explicitly call ecr:GetAuthorizationToken to generate a Docker login token, as the authentication mechanism is handled at a lower level by the AWS SDK integrated into the Kubelet's image pulling logic. However, including it causes no harm and might be useful in some debugging scenarios or with older EKS versions.

Q2: Can I reuse an existing IAM role for an EKS Service Account?

Yes, you can reuse an existing IAM role. However, you MUST ensure that the existing IAM role's trust policy is updated to include the EKS OIDC provider and the specific conditions for the service account (oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>:sub and aud). Without the correct trust relationship, the Kubernetes Service Account will not be able to assume the role.

Q3: What if I have checked all IAM and network configurations, and ImagePullBackOff still persists?

If you've meticulously followed all steps and the issue persists, consider these advanced checks:

• Kubelet Logs: SSH into the EKS worker node and inspect the Kubelet logs (e.g., journalctl -u kubelet or check /var/log/pods) for more detailed errors during image pull attempts.
• AWS STS Service Status: Verify that the AWS Security Token Service (STS) endpoint your cluster is configured to use is healthy and accessible. IRSA relies on STS.
• Pod Annotations vs. Deployment Annotations: Ensure that if you're deploying with a Deployment, the serviceAccountName is set in the Pod's template spec, and the Service Account itself has the IRSA annotation, not directly on the Deployment object.
• AWS Region Mismatch: Double-check that all components (EKS cluster, ECR repository, IAM role) are in the same AWS region, and image URLs reflect this.
• Temporary Credentials Expiration: While less common with IRSA, ensure no hardcoded or stale temporary credentials are inadvertently being used somewhere.