Tech Note: Always backup your configuration files before applying any changes to production environments.

Troubleshooting AWS EKS Pod Eviction Due to Node Pressure (DiskPressure)

Understanding and resolving pod evictions due to DiskPressure in AWS EKS is crucial for maintaining application stability and performance. This comprehensive guide provides a deep dive into the symptoms, root causes, and a step-by-step resolution process, coupled with best practices to prevent future occurrences. As a Senior Cloud Solution Architect, optimizing resource utilization and ensuring resilient Kubernetes operations is paramount.

Symptom Analysis & Root Causes

DiskPressure is a Kubernetes node condition indicating that the node's disk is running low on available space for either the image filesystem or the node's root filesystem. When this condition is met, the Kubelet will begin evicting pods to reclaim disk space, leading to service disruptions.

Key Symptoms of DiskPressure

• Pod Evictions: Pods on affected nodes transition to Evicted status.
• Node Conditions: Executing kubectl describe node <node-name> will show DiskPressure: True under the Conditions section.
• Event Logs: Kubernetes events will report messages like "NodeHasDiskPressure" or "Failed to allocate new cgroup" for containers.
• Application Instability: Unresponsive applications, increased latency, or downtime due to evicted pods.
• Unable to Schedule Pods: New pods may fail to schedule on the affected node due to the DiskPressure taint.

Common Root Causes

• Excessive Log Accumulation: Application logs, container runtime logs (e.g., containerd/docker), or system logs (journald) consuming a large portion of the disk.
• Ephemeral Storage Mismanagement: Application pods generating large temporary files, caches, or process data without proper cleanup, exceeding their ephemeral storage limits (if defined) or simply overwhelming the node's disk.
• Large Container Images and Image Caching: Many large container images pulled over time accumulate in the image filesystem, especially without effective garbage collection or aggressive image pruning.
• Under-provisioned Disk Size: The underlying EC2 instance type (or custom AMI) might have a root volume too small for the workload and Kubernetes overhead.
• Kubelet Eviction Thresholds: Default Kubelet configuration for eviction thresholds might be too aggressive or misconfigured, leading to premature evictions.
• Volume Mounts Issues: Misconfigured persistent volumes or a large number of ephemeral emptyDir volumes that grow unbounded.
• Containerd/Docker OverlayFS Bloat: The storage driver's layers can accumulate significant data over time if not managed.

Step-by-Step Resolution Guide

Step 1: Identify Affected Nodes and Pods

First, pinpoint which nodes are experiencing DiskPressure and which pods are being evicted.

kubectl get nodes kubectl describe node <node-name> kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=<node-name> kubectl get pods --all-namespaces -o wide --field-selector status.phase=Evicted

Look for DiskPressure: True in the node description and eviction events.

Step 2: SSH into the Affected Node

Obtain the public IP address of the affected node (e.g., from kubectl get nodes -o wide or AWS EC2 console) and SSH into it. For EKS nodes, typically use ec2-user with your EC2 key pair.

ssh -i <your-key.pem> ec2-user@<node-public-ip>

Step 3: Analyze Disk Usage on the Node

Once logged in, use standard Linux commands to identify which directories are consuming the most disk space.

# Check overall disk usage df -h # Identify large directories, especially /var/lib/kubelet, /var/log, /var/lib/containerd or /var/lib/docker sudo du -sh /* | sort -rh sudo du -sh /var/* | sort -rh sudo du -sh /var/lib/kubelet/* | sort -rh sudo du -sh /var/lib/containerd/* | sort -rh # If using containerd sudo du -sh /var/lib/docker/* | sort -rh # If using Docker

Pay close attention to /var/log (system and container logs), /var/lib/kubelet (pods, volumes, image blobs), and /var/lib/containerd or /var/lib/docker (container images and writable layers).

Step 4: Clean Up Unnecessary Files and Data

Based on your findings in Step 3, proceed with targeted cleanup.

Clean Up Logs:

Clear old journald logs and application logs.

# Check journald disk usage sudo journalctl --disk-usage # Reduce journald logs to a specific size (e.g., 1GB) sudo journalctl --vacuum-size=1G # Remove old compressed logs sudo find /var/log -name "*.gz" -delete # For specific application logs, you might need to find and remove them or configure log rotation # Example: Delete logs older than 7 days in a specific directory sudo find /var/log/my-app -type f -name "*.log" -mtime +7 -delete

Clean Up Container Runtime Data (Caution Required):

Manually cleaning container runtime data should be done with extreme care, especially in production. Kubernetes is designed to handle this, but manual intervention can provide immediate relief.

# List container images (for containerd) sudo crictl images # List containers (for containerd) sudo crictl ps -a # Manually remove unused images (use with caution, Kubelet manages this) # Example: Remove an old image by ID (replace with actual image ID) sudo crictl rmi <image-id> # For Docker (if used as runtime) - prune all unused Docker data (images, containers, networks, build cache) # This is aggressive and might impact running containers if not careful. # It's generally better to let Kubelet manage this. # sudo docker system prune -a --volumes

Important: After manual cleanup, restart kubelet to re-evaluate disk space.

sudo systemctl restart kubelet

Step 5: Adjust Kubelet Eviction Thresholds (Temporary/Advanced)

While not a permanent solution, you can temporarily adjust Kubelet's eviction thresholds to buy time or tolerate slightly higher disk usage. This involves modifying the Kubelet configuration on the node.

Find the Kubelet configuration file, typically at /etc/kubernetes/kubelet/kubelet-config.json or passed via flags in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. For EKS optimized AMIs, parameters are often passed via the /etc/eks/bootstrap.sh script.

You would look for and modify parameters like:

# Example Kubelet config snippet kubelet --eviction-hard='imagefs.available<5%,nodefs.available<5%' \ --eviction-soft='imagefs.available<10%,nodefs.available<10%' \ --eviction-soft-grace-period='nodefs.available=1m' \ --image-gc-high-threshold=85 \ --image-gc-low-threshold=80

Note: Modifying these settings directly on production nodes is risky. It's better to update your EKS node group's launch template or user data to persist these changes across reboots and scale-outs.

Step 6: Scale Up or Vertically Scale Nodes

If the disk pressure is a recurring issue, consider increasing the disk size of your EKS worker nodes or scaling out your node group.

• Increase Disk Size: Modify the launch template used by your EKS node group to provision larger root volumes (e.g., from 20GB to 50GB or 100GB). This often requires rolling out new nodes.
• Add More Nodes: Utilize Kubernetes Cluster Autoscaler or Karpenter to automatically add more nodes to your cluster, distributing the workload and disk consumption.
• Use Larger Instance Types: Some EC2 instance types come with larger default root volumes or NVMe SSD ephemeral storage (though ephemeral storage is generally not for persistence).

Step 7: Monitor and Verify

After applying fixes, continuously monitor the node and cluster status.

kubectl describe node <node-name> # Verify DiskPressure: False kubectl get events

Ensure no new eviction events occur and that the node returns to a healthy state.

Best Practices for Prevention & Performance Optimization

Proactive Monitoring

• AWS CloudWatch: Monitor EC2 instance disk utilization metrics (e.g., DiskReadBytes, DiskWriteBytes, DiskUsage if agent installed).
• Prometheus & Grafana: Deploy a robust monitoring stack to collect Kubelet metrics (kubelet_node_config_error, node_filesystem_avail_bytes, node_filesystem_size_bytes) and create alerts for low disk space.

Resource Management

• Ephemeral Storage Limits: Define requests and limits for ephemeral-storage in your pod specifications. This prevents a single misbehaving pod from consuming all node disk space.
  resources: limits: memory: "1Gi" cpu: "500m" ephemeral-storage: "5Gi" # Limit ephemeral storage to 5GB requests: memory: "512Mi" cpu: "250m" ephemeral-storage: "1Gi" # Request 1GB of ephemeral storage
• Log Management: Implement centralized logging (e.g., AWS CloudWatch Logs, Fluent Bit to S3/ELK stack) and configure log rotation on nodes to prevent local disk bloat.
• Smaller Container Images: Use minimal base images (e.g., Alpine, distroless) to reduce the overall disk footprint of container images.
• Image Garbage Collection: Ensure Kubelet's image garbage collection is effectively configured (--image-gc-high-threshold, --image-gc-low-threshold).

Node Group and Autoscaling

• Appropriate Node Size: Select EC2 instance types with sufficient root volume sizes for your workloads and Kubernetes overhead. Consider customizing the root volume size in your EKS node group's launch template.
• Cluster Autoscaler/Karpenter: Leverage these tools to automatically scale your node groups based on resource demand, ensuring sufficient capacity and preventing single-node saturation.

Storage Strategy

• Persistent Volumes: For data that needs to persist across pod restarts or that consumes significant disk space, use Persistent Volumes (PVs) backed by AWS EBS or EFS, rather than relying on ephemeral node storage.

Frequently Asked Questions (FAQs)

Q1: What is the difference between DiskPressure and MemoryPressure?

A: DiskPressure indicates that the node's disk space is critically low, primarily for the image filesystem or node's root filesystem, leading Kubelet to evict pods to free up disk. MemoryPressure, on the other hand, means the node is running low on available memory, which can also trigger Kubelet to evict pods based on their memory usage to reclaim RAM. Both are node conditions that affect pod scheduling and stability.

Q2: Can I disable pod eviction entirely to prevent service disruption?

A: While you can technically set Kubelet's eviction thresholds very high or disable eviction signals (--eviction-hard=""), it's strongly discouraged. Disabling evictions means that if a node runs out of critical resources like disk space or memory, it can become completely unstable, unresponsive, and even crash, leading to much worse service disruptions. Evictions are a protective mechanism to keep the node healthy and allow workloads to be rescheduled elsewhere. The correct approach is to address the root cause of resource exhaustion and manage resources effectively.

Q3: How do I calculate appropriate ephemeral storage limits for my pods?

A: Calculating ephemeral storage limits requires understanding your application's behavior.

• Monitor Usage: Deploy your application without explicit ephemeral storage limits initially (in a testing environment) and monitor its disk usage within the container using tools like kubectl top pod <pod-name> --containers (though this mainly shows CPU/Mem) or by SSHing into a running container and using du -sh /tmp (or wherever it writes temporary data).
• Analyze Logs/Caches: Identify how much space application logs, temporary files, and caches consume during peak operations.
• Buffer: Always add a buffer (e.g., 20-30%) above the observed peak usage to account for unforeseen spikes or growth.
• Iterate: Start with a reasonable limit, observe, and adjust as needed. Remember, setting limits too low can lead to pod evictions if the application exceeds them.