Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing Kubernetes Pods Stuck in ContainerCreating with AWS EFS PVC Errors on EKS

As a Senior Cloud Solution Architect and Software Engineer, I frequently encounter complex issues within Kubernetes environments. One common and particularly frustrating problem on Amazon Elastic Kubernetes Service (EKS) involves pods getting perpetually stuck in the ContainerCreating state, often accompanied by persistent volume claim (PVC) errors related to AWS Elastic File System (EFS). This guide provides a comprehensive, SEO-optimized technical breakdown and a step-by-step troubleshooting manual to diagnose and resolve such issues.

Understanding the ContainerCreating State and EFS PVC Errors

The ContainerCreating status indicates that Kubernetes has successfully scheduled a pod to a node, but it's unable to start the containers within that pod. When AWS EFS is involved as the storage backend via a PersistentVolumeClaim (PVC), this often points to issues with the EFS CSI (Container Storage Interface) driver, underlying AWS infrastructure, or misconfigurations that prevent the EFS volume from being correctly mounted to the EKS worker node.

Symptom Analysis & Root Causes

When a pod is stuck in ContainerCreating due to EFS PVC errors, you'll typically observe specific events and log messages. Identifying these patterns is crucial for effective troubleshooting.

Common Symptoms:

• Pods consistently show Status: ContainerCreating.
• Running kubectl describe pod <pod-name> reveals events such as:
  • FailedAttachVolume: Unable to attach volume "pvc-..." to node "ip-..."
  • FailedMount: MountVolume.SetUp failed for volume "pvc-..." : rpc error: code = Internal desc = EFS: mount target not found or EFS: failed to create access point or mount.nfs: access denied by server while mounting
  • Warning FailedScheduling: 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims.
• EFS CSI driver pod logs show errors related to IAM permissions, network connectivity, or file system access.

Key Root Causes:

Several factors can contribute to these issues. Understanding them helps in targeted troubleshooting:

1. EFS CSI Driver Misconfiguration or Absence: The AWS EFS CSI driver must be correctly installed and configured on your EKS cluster to handle EFS volume provisioning and mounting. If it's missing, outdated, or misconfigured, volume operations will fail.
2. IAM Permissions Insufficiency: The IAM role associated with the EFS CSI driver's Kubernetes service account might lack the necessary permissions to perform EFS actions (e.g., elasticfilesystem:DescribeMountTargets, elasticfilesystem:CreateAccessPoint, elasticfilesystem:ClientMount).
3. Security Group Misconfiguration:
   • The security groups attached to your EKS worker nodes might not allow outbound NFS traffic (port 2049) to the EFS mount targets.
   • The security groups attached to your EFS mount targets might not allow inbound NFS traffic (port 2049) from the EKS worker node security groups.
4. Networking and EFS Mount Target Issues:
   • EFS mount targets might not exist in all subnets where your EKS worker nodes are running, or they might be in private subnets inaccessible to worker nodes (or vice-versa).
   • Network ACLs (NACLs) preventing traffic between worker nodes and EFS.
5. Incorrect PersistentVolume (PV) / PersistentVolumeClaim (PVC) Configuration:
   • Mismatched storageClassName between PVC and a valid EFS StorageClass.
   • Incorrect EFS File System ID specified in the PV or dynamically provisioned PVC.
   • Unsupported accessModes (EFS only supports ReadWriteMany for dynamic provisioning).
6. EFS File System Policy: A restrictive EFS file system policy might be preventing access from the IAM role used by the CSI driver or pods.
7. KMS Key Permissions (if EFS encrypted): If your EFS file system is encrypted with a custom AWS Key Management Service (KMS) key, the IAM role used by the EFS CSI driver might not have permission to decrypt data using that key.

Step-by-Step Resolution Guide

Follow these steps sequentially to diagnose and resolve EFS PVC errors causing pods to be stuck in ContainerCreating.

Step 1: Verify EFS CSI Driver Installation and Status

Ensure the EFS CSI driver is installed and its pods are running correctly.

Check for driver pods (usually in the kube-system namespace):

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-efs-csi-driver

Expected output: pods in Running status. If not, inspect pod logs for errors:

kubectl logs -n kube-system <efs-csi-controller-pod-name> kubectl logs -n kube-system <efs-csi-node-pod-name>

Verify the CSIDriver object:

kubectl get CSIDriver efs.csi.aws.com

And the StorageClass:

kubectl get storageclass

Ensure you have a StorageClass similar to this (replace with your actual StorageClass name):

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: efs-sc provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: fs-0123456789abcdef0 # Optional for dynamic provisioning if an AP is created per PVC directoryPerms: "777" gidRangeStart: "1000" # Optional gidRangeEnd: "2000" # Optional reclaimPolicy: Retain volumeBindingMode: Immediate

If the driver is not installed, follow the official AWS EKS documentation to install it using Helm.

Step 2: Validate IAM Permissions for EFS CSI Driver

The EFS CSI driver's service account requires specific IAM permissions. EKS uses IAM Roles for Service Accounts (IRSA) for this. Find the service account used by the driver:

kubectl get sa -n kube-system efs-csi-controller-sa -o yaml | grep "arn:aws:iam"

Note the IAM role ARN. Go to the AWS IAM console, find this role, and ensure it has a policy attached with at least these permissions:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "elasticfilesystem:DescribeAccessPoints", "elasticfilesystem:DescribeFileSystems", "elasticfilesystem:DescribeMountTargets", "ec2:DescribeAvailabilityZones" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "elasticfilesystem:CreateAccessPoint" ], "Resource": "*", "Condition": { "StringEquals": { "aws:RequestTag/eks.amazonaws.com/cluster-name": "<your-cluster-name>" } } }, { "Effect": "Allow", "Action": [ "elasticfilesystem:DeleteAccessPoint" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/eks.amazonaws.com/cluster-name": "<your-cluster-name>" } } } ] }

For EFS file systems encrypted with KMS, additional kms:Decrypt permissions on the specific KMS key might be required.

Step 3: Examine Security Groups Configuration

Networking is a common pitfall. Ensure NFS traffic (port 2049) is allowed between EKS worker nodes and EFS mount targets.

1. EKS Worker Node Security Groups:

   Identify the security groups attached to your EKS worker nodes. These need an outbound rule allowing TCP traffic on port 2049 to the security groups of your EFS mount targets.

2. EFS Mount Target Security Groups:

   Identify the security groups attached to your EFS mount targets. These need an inbound rule allowing TCP traffic on port 2049 from the security groups of your EKS worker nodes.

A good practice is to create a dedicated security group for EFS mount targets and reference it in the worker node security groups, and vice-versa.

Step 4: Check EFS Mount Targets & Networking

Ensure EFS mount targets exist in all relevant subnets where your EKS worker nodes are scheduled. EFS is a regional service, but access is via mount targets in specific AZs. If a pod lands on a node in an AZ without an EFS mount target, it cannot mount the volume.

Verify mount target presence in the AWS EFS console. For optimal availability, provision mount targets in every AZ used by your EKS cluster.

You can test connectivity from an EKS worker node (SSH into it) to an EFS mount target IP:

sudo yum install -y nfs-utils # Or apt-get install nfs-common sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <EFS_FILE_SYSTEM_ID>.efs.<REGION>.amazonaws.com:/ /mnt/test # To test specific IP nc -vz <EFS_MOUNT_TARGET_IP> 2049

If nc fails, it's a network/security group issue. If mount fails, check EFS policy or permissions.

Step 5: Review PersistentVolumeClaim (PVC) and PersistentVolume (PV) Configuration

Inspect your PVC and the dynamically created PV to ensure they are correctly configured.

kubectl get pvc <your-pvc-name> -o yaml kubectl describe pvc <your-pvc-name> kubectl get pv <corresponding-pv-name> -o yaml kubectl describe pv <corresponding-pv-name>

Look for:

• status: Bound for the PVC.
• The storageClassName in your PVC matching the name of your EFS StorageClass.
• In the PV, ensure spec.csi.volumeHandle is the correct EFS File System ID (e.g., fs-0123456789abcdef0) or an EFS Access Point ID.
• accessModes: ReadWriteMany is typically used for EFS.

Step 6: Verify EFS File System Policy

A restrictive EFS file system policy can block access. Navigate to the AWS EFS console, select your file system, and check the "File system policy" section. Ensure it allows root access from the EFS CSI driver's IAM role or appropriate client access.

A common policy allowing clients to connect without IAM authorization (but relies on security groups) looks like:

{ "Version": "2012-10-17", "Id": "efs-policy-example", "Statement": [ { "Sid": "EfsAllowAllAccess", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "elasticfilesystem:ClientMount", "elasticfilesystem:ClientWrite", "elasticfilesystem:ClientRootAccess" ], "Resource": "<arn-of-your-efs-filesystem>" } ] }

If you are using IAM authorization for EFS, ensure the principal includes the IAM role of your EFS CSI driver and your EKS worker nodes.

Step 7: KMS Key Permissions (if applicable)

If your EFS is encrypted with a custom KMS key, the EFS CSI driver's IAM role and potentially the EKS worker node role need kms:Decrypt permissions on that key. Go to AWS KMS, find your key, and check its key policy. Add the EFS CSI driver's IAM role as a principal with kms:Decrypt action.

Step 8: Restart Affected Pods/Deployments

After applying any configuration changes (IAM, Security Groups, EFS Policy, etc.), force the affected pods to restart so they can pick up the new configurations.

kubectl rollout restart deployment <your-deployment-name> -n <namespace> # Or delete individual pods to let the controller recreate them kubectl delete pod <pod-name> -n <namespace>

Best Practices for Prevention & Performance Optimization

Preventing these issues is better than fixing them. Adopt these best practices for a more robust and performant EKS-EFS integration:

• Infrastructure as Code (IaC): Always manage your EKS cluster, EFS file systems, mount targets, security groups, and IAM roles using IaC tools like AWS CloudFormation or Terraform. This ensures consistency and reduces manual error.
• Regular EFS CSI Driver Updates: Keep your EFS CSI driver updated to the latest stable version. AWS frequently releases updates with bug fixes and performance improvements.
• Least Privilege IAM: Adhere strictly to the principle of least privilege for the EFS CSI driver's IAM role. Grant only the necessary permissions required for EFS operations.
• Robust Networking Design:
  • Create EFS mount targets in all Availability Zones where your EKS worker nodes are provisioned.
  • Design security groups meticulously, explicitly allowing NFS traffic (port 2049) between worker nodes and EFS mount targets.
• EFS Performance Modes & Throughput: Choose the appropriate EFS performance mode (General Purpose vs. Max I/O) and throughput mode (Bursting vs. Provisioned) based on your application's requirements. Misconfigured performance can lead to application slowdowns, though not directly to ContainerCreating issues.
• Monitoring & Alerting: Implement comprehensive monitoring for both EKS (pod statuses, events, controller logs) and EFS (CloudWatch metrics like ClientConnections, PermittedThroughput, BurstCreditBalance). Set up alerts for critical thresholds.
• Dedicated StorageClass: Define a clear, dedicated StorageClass for EFS to enforce consistent provisioning settings across your cluster.

Frequently Asked Questions (FAQs)

Q1: What does it mean for a Pod to be in the "ContainerCreating" state?

A pod in the ContainerCreating state signifies that Kubernetes has successfully scheduled the pod to a worker node, but one or more containers within that pod are unable to start. This often occurs when a container image cannot be pulled, necessary volumes cannot be mounted (as with EFS PVC errors), required secrets/configmaps are missing, or there are resource constraints preventing the container runtime from initializing.

Q2: How do I efficiently troubleshoot logs for the EFS CSI driver?

To efficiently troubleshoot, first identify the EFS CSI driver pods in the kube-system namespace using kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-efs-csi-driver. Then, use kubectl logs -n kube-system <pod-name> to view logs for each controller and node pod. For real-time monitoring, add the -f flag (kubectl logs -f -n kube-system <pod-name>). Look for keywords like "error," "failed," "permission denied," "mount," or "EFS" in the logs for specific clues.

Q3: Can AWS EFS be used across multiple Availability Zones in EKS?

Yes, AWS EFS is a regional service, meaning your file system data is redundantly stored across multiple Availability Zones within an AWS Region. To allow EKS worker nodes in different AZs to access the same EFS file system, you must create an EFS mount target in each Availability Zone where your EKS worker nodes reside. The EFS CSI driver will automatically use the appropriate mount target for a pod based on its residing AZ, providing high availability and shared access across your cluster.

---END_GUIDE---