Tech Note: Always backup your configuration files before applying any changes to production environments.
Resolving ImagePullBackOff for Private ECR Images in AWS EKS with IAM Roles for Service Accounts (IRSA)
The ImagePullBackOff error is a common frustration for developers and operations teams deploying applications on Kubernetes. While its occurrence can stem from various issues, when working with AWS Elastic Kubernetes Service (EKS) and private Elastic Container Registry (ECR) images, it often points to an authorization problem. This comprehensive guide will walk you through diagnosing and resolving ImagePullBackOff specifically when your EKS pods fail to pull private ECR images, leveraging the secure and recommended approach of IAM Roles for Service Accounts (IRSA).
Symptom Analysis & Root Causes
The ImagePullBackOff Error
When an EKS pod enters an ImagePullBackOff state, it means that Kubernetes tried to pull a container image for the pod, failed, and will continuously retry with increasing back-off delays. You'll typically see this status when checking pod events or descriptions.
To quickly check for this, use:
kubectl get pods
kubectl describe pod <pod-name> -n <namespace>
Look for events like Failed to pull image "aws_account_id.dkr.ecr.region.amazonaws.com/my-private-repo:latest": rpc error: code = Unknown desc = Error response from daemon: Get "https://aws_account_id.dkr.ecr.region.amazonaws.com/v2/my-private-repo/manifests/latest": no basic auth credentials or Failed to pull image "aws_account_id.dkr.ecr.region.amazonaws.com/my-private-repo:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for aws_account_id.dkr.ecr.region.amazonaws.com/my-private-repo, repository does not exist or may require 'docker login'.
Common Root Causes for ECR-related ImagePullBackOff
- Insufficient IAM Permissions: The most frequent culprit. The IAM role associated with your EKS nodes (or more precisely, the IAM Role for Service Account) lacks the necessary permissions to pull images from ECR. This often means missing
ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, and ecr:BatchCheckLayerAvailability actions, or access to the ecr:GetAuthorizationToken action.
- Incorrect ECR Repository Policy: While less common for pull failures, a restrictive ECR repository policy can block access even if the IAM user/role has permissions. Ensure the repository policy doesn't explicitly deny your EKS nodes or IRSA.
- Misconfigured IAM Roles for Service Accounts (IRSA): If using IRSA, misconfigurations like an incorrect
serviceAccountName in your deployment, a missing OIDC provider for the EKS cluster, or incorrect IAM trust policy for the service account role can lead to authentication failures.
- Network Connectivity Issues: The EKS nodes might not have network access to the ECR endpoints. This can happen if security groups, NACLs, or routing tables block traffic, or if a VPC endpoint for ECR is misconfigured or missing in a private-only subnet setup.
- Image Not Found or Wrong Tag: The image name or tag specified in the pod definition might be incorrect or not exist in the ECR repository. This leads to a different type of pull error but is worth checking.
Step-by-Step Resolution Guide: Leveraging IRSA for ECR Access
Using IAM Roles for Service Accounts (IRSA) is the most secure and recommended way to grant AWS permissions to workloads running in EKS. It allows you to associate an IAM role directly with a Kubernetes Service Account, which your pods then use. This eliminates the need to grant broad permissions to the EKS node instance profile.
Pre-requisites
- An operational AWS EKS Cluster.
- kubectl configured to communicate with your EKS cluster.
- aws CLI installed and configured with appropriate permissions.
- eksctl installed (highly recommended for IRSA management).
- jq for JSON parsing (optional but useful).
- Your EKS cluster must have an OIDC provider enabled.
Step 1: Verify EKS Cluster OIDC Provider
IRSA relies on an OpenID Connect (OIDC) provider for your EKS cluster. If you created your EKS cluster with eksctl, it's likely already enabled. Otherwise, you might need to enable it manually.
Check if an OIDC provider exists for your cluster:
aws eks describe-cluster --name <your-cluster-name> --query "cluster.identity.oidc.issuer" --output text
The output should be a URL like https://oidc.eks.<region>.amazonaws.com/id/<OIDC_ID>.
If it doesn't exist, you can create it using eksctl:
eksctl utils associate-iam-oidc-provider --cluster <your-cluster-name> --approve
Step 2: Create an IAM Policy for ECR Read-Only Access
This policy grants the necessary permissions to pull images from ECR. Apply the principle of least privilege, granting access only to specific repositories if possible.
Create a file named ecr-pull-policy.json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:GetAuthorizationToken"
],
"Resource": "*"
}
]
}
Note: For production environments, restrict the Resource to specific ECR repository ARNs instead of "*". E.g., "arn:aws:ecr:<region>:<account-id>:repository/<repo-name>". However, ecr:GetAuthorizationToken usually requires "*" for its resource.
Create the IAM policy:
aws iam create-policy \
--policy-name ECRImagePullPolicy \
--policy-document file://ecr-pull-policy.json
Note down the Arn of the created policy.
Step 3: Create an IAM Service Account (IRSA) for your EKS Workload
Now, create a Kubernetes Service Account and associate it with an IAM role that has the ECR pull policy. eksctl simplifies this significantly.
eksctl create iamserviceaccount \
--cluster=<your-cluster-name> \
--namespace=<your-namespace> \
--name=<your-service-account-name> \
--attach-policy-arn=arn:aws:iam::<account-id>:policy/ECRImagePullPolicy \
--override-existing-serviceaccounts \
--approve
This command will:
- Create a Kubernetes Service Account named
<your-service-account-name> in <your-namespace>.
- Create an IAM Role with the specified policy.
- Configure the IAM Role's trust policy to allow the EKS OIDC provider to assume it, authenticated via the Kubernetes Service Account.
- Annotate the Kubernetes Service Account with the ARN of the new IAM Role.
Step 4: Configure Your Kubernetes Deployment to Use the Service Account
Update your Kubernetes Deployment, DaemonSet, or StatefulSet to use the newly created Service Account.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-deployment
namespace: <your-namespace>
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
serviceAccountName: <your-service-account-name> # THIS IS CRITICAL
containers:
- name: my-app-container
image: <aws_account_id>.dkr.ecr.<region>.amazonaws.com/my-private-repo:latest
ports:
- containerPort: 80
Apply the updated deployment:
kubectl apply -f your-deployment.yaml
Step 5: Verify ECR Image Pull
Monitor the status of your pods. They should now be able to pull images successfully.
kubectl get pods -n <your-namespace>
# Wait until the pod status is Running
kubectl describe pod <pod-name> -n <your-namespace>
Check the events in the kubectl describe pod output. You should see successful image pull events.
Troubleshooting Checklist and Advanced Tips
Initial Checks
- Pod Status & Events: Always start with
kubectl describe pod <pod-name>. Look for specific error messages under the 'Events' section. Common messages include "pull access denied," "repository does not exist," or "no basic auth credentials."
- ECR Image Availability: Double-check the full ECR image URI and tag in your deployment.yaml. Does the image and tag actually exist in the specified ECR repository?
- Network Connectivity:
- Ensure your EKS worker nodes can reach ECR endpoints. If using private subnets, confirm that VPC Endpoints for ECR (API, DKR) are configured correctly and associated with the right subnets and security groups.
- Check security group rules for EKS nodes to allow outbound HTTPS (port 443) traffic to ECR.
Deep Dive into IRSA Configuration
ECR Repository Policy
While rare for denying basic pull access, a restrictive ECR repository policy can override IAM user/role permissions. If you suspect this, check the repository policy:
aws ecr get-repository-policy --repository-name <your-repo-name>
Ensure there are no explicit "Deny" statements that would block the IAM Role associated with your service account. If you need to set a policy (e.g., for cross-account access), it would look like this (example):
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPushPull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id-of-irsa>:role/<iam-role-name-for-irsa>"
},
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:BatchGetImage",
"ecr:CompleteLayerUpload",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"
]
}
]
}
aws ecr set-repository-policy \
--repository-name <your-repo-name> \
--policy-text file://your-ecr-repo-policy.json
Best Practices for Prevention & Performance Optimization
Principle of Least Privilege
Always grant only the permissions necessary for your workloads to function. Instead of Resource: "*", specify exact ECR repository ARNs for pull actions where possible.
ECR Repository Policies
Use ECR repository policies for finer-grained control, especially in cross-account scenarios or when needing to restrict access to specific IAM entities for particular repositories.
Image Tagging and Management
Maintain consistent and clear image tagging conventions. Avoid using :latest in production; prefer immutable tags (e.g., commit hashes, version numbers) to ensure predictable deployments and easier rollbacks.
Network Connectivity & VPC Endpoints
For enhanced security and performance, especially in private subnets, configure VPC Endpoints for ECR (both the API and DKR endpoints). This keeps traffic within the AWS network and avoids routing through the public internet. Ensure associated security groups allow traffic.
Regular Monitoring and Alerting
Implement monitoring for pod events and ECR API calls. CloudWatch Logs for EKS and ECR can provide valuable insights into authentication and authorization failures, allowing for proactive issue detection.
Frequently Asked Questions (FAQs)
Q1: What if I don't use IRSA and my nodes are in private subnets?
If you're not using IRSA, your EKS worker nodes' instance profile IAM role must have the ECR pull permissions. Additionally, if in private subnets, you absolutely need VPC Endpoints for ECR (ecr.api and ecr.dkr) to allow nodes to pull images without public internet access.
Q2: How can I debug ImagePullBackOff more effectively?
Beyond kubectl describe pod, consider temporarily SSHing into an affected EKS node (if allowed) and trying to perform a docker login to ECR manually. This can help isolate if it's a node-level network or credential issue versus a Kubernetes/IRSA configuration problem.
# On the EKS worker node
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
docker pull <aws_account_id>.dkr.ecr.<region>.amazonaws.com/my-private-repo:latest
Q3: Is it secure to grant ECR pull permissions directly to EKS node instance roles?
While it works, it's generally considered less secure than using IRSA. Granting permissions to the node's instance profile means any pod running on that node *could* potentially assume those permissions, violating the principle of least privilege. IRSA provides granular, per-service-account permissions, isolating roles to specific workloads.
By systematically following these steps and understanding the underlying mechanisms of IRSA and ECR, you can efficiently diagnose and resolve ImagePullBackOff errors, ensuring your applications deploy smoothly on AWS EKS.