Tech Note: Always backup your configuration files before applying any changes to production environments.

Troubleshooting Kubernetes Pod Eviction Due to Node Pressure on AWS EKS Managed Node Groups

As a Senior Cloud Solution Architect and Software Engineer, navigating the complexities of Kubernetes operations on AWS EKS is a daily endeavor. One of the most common yet challenging issues encountered is pod eviction due to node pressure. This comprehensive guide and troubleshooting manual will delve into the intricacies of identifying, diagnosing, and resolving pod evictions on AWS EKS Managed Node Groups, ensuring your applications remain stable and performant.

Symptom Analysis & Root Causes

Understanding the symptoms and underlying causes of pod evictions is crucial for effective troubleshooting. Kubernetes proactively evicts pods from nodes that are experiencing resource contention to maintain node stability and overall cluster health.

Symptoms of Node Pressure Eviction

• Pods showing Evicted status when running kubectl get pods.
• Events logged against pods or nodes indicating Evicted, NodeHasDiskPressure, NodeHasMemoryPressure, or NodeHasPidPressure.
• Applications experiencing intermittent outages or slow response times.
• High resource utilization observed on specific nodes (CPU, Memory, Disk I/O, PIDs).
• kubelet logs on the affected nodes showing warnings about resource thresholds.

Common Root Causes of Node Pressure Eviction

Node pressure in Kubernetes primarily manifests in three forms, each with distinct causes:

• Memory Pressure:
  • OOMKill by Linux Kernel: Processes inside pods exceeding their memory limits, leading to the Linux OOM killer terminating them.
  • Insufficient Node Memory: The node itself running out of physical memory, even if individual pods are within limits. This can be exacerbated by non-Kubernetes processes or excessive memory overhead from the OS, container runtime, or kubelet.
  • Memory Leaks: Applications or sidecar containers with memory leaks that progressively consume more memory.
  • Misconfigured Resource Limits: Pods configured with unrealistically low memory limits, or the sum of all pod memory requests/limits exceeding node capacity.
• Disk Pressure:
  • Ephemeral Storage Exhaustion: Pods writing excessive logs, temporary files, or cached data, consuming the node's ephemeral storage (root filesystem).
  • Container Image Cache: Large number of container images or large image layers accumulating on the node, consuming disk space.
  • Volume Issues: Persistent Volumes (PVs) or Persistent Volume Claims (PVCs) consuming excessive disk space on the underlying storage, though direct node disk pressure is usually about ephemeral storage.
  • Logging Spikes: Applications generating a high volume of logs without proper log rotation or offloading mechanisms.
• PID Pressure:
  • Process ID Exhaustion: A node running out of available Process IDs (PIDs), preventing new processes from being created. This can happen if many containers are running on the node, or if processes within containers fork many child processes without proper cleanup.
  • Zombie Processes: Unreaped child processes accumulating on the node.
  • High Process Churn: Applications or systems that rapidly create and destroy processes.

Step-by-Step Resolution Guide

Follow these steps to diagnose and resolve pod evictions due to node pressure on your AWS EKS Managed Node Groups.

1. Initial Assessment: Identify Evicted Pods and Node Events

Begin by listing all pods, filtering for those in an Evicted state, and inspecting their events.

kubectl get pods --all-namespaces | grep Evicted kubectl describe pod <evicted-pod-name> -n <namespace>

Look for event messages like The node was low on resource: memory, The node was low on resource: ephemeral-storage, or The node was low on resource: pid. This will pinpoint the type of pressure.

2. Identify the Affected Nodes and Their Status

Determine which nodes are experiencing pressure. Nodes will typically show a Ready,MemoryPressure, Ready,DiskPressure, or Ready,PIDPressure status.

kubectl get nodes kubectl describe node <node-name>

In the kubectl describe node output, check the Conditions section for pressure alerts and the Allocated resources section to see what pods are consuming on the node.

3. Addressing Memory Pressure

Memory pressure is a common culprit. Here's how to tackle it:

• Analyze Pod Resource Usage: Use monitoring tools (Prometheus/Grafana, Datadog, CloudWatch Container Insights) to identify memory-hungry pods on the affected node.
• Adjust Pod Resource Requests and Limits: Increase memory limits for pods frequently evicted, but also ensure requests are set realistically to allow effective scheduling.
apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: spec: containers: - name: my-container image: my-image:latest resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" # Increase this value if necessary cpu: "500m"
• Scale Up Node Group: If many nodes are under memory pressure, your EKS Managed Node Group might be undersized. Increase the desired size or maximum size of the ASG backing your node group.
• Optimize Application Memory Usage: Profile your applications for memory leaks or inefficient memory allocation.
• Review Kubelet Configuration (Advanced): For extreme cases, you might adjust kubelet's eviction thresholds. However, for EKS Managed Node Groups, AWS manages much of the underlying OS and kubelet configuration, so this is less common and should be approached with caution.

4. Addressing Disk Pressure (Ephemeral Storage)

Disk pressure often stems from ephemeral storage consumption.

• Identify High Disk Usage Pods:
  # Get usage metrics for ephemeral storage (if metrics server is deployed) kubectl top pod --all-namespaces --containers --sort-by='ephemeral-storage'

  Alternatively, SSH into the node and use du -sh /var/lib/kubelet/pods/<pod-uid> to identify actual disk usage.

• Set Ephemeral Storage Limits: Configure ephemeral-storage requests and limits for pods. This prevents a single misbehaving pod from consuming all node disk space.
apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: spec: containers: - name: my-container image: my-image:latest resources: requests: ephemeral-storage: "1Gi" limits: ephemeral-storage: "5Gi" # Set a reasonable limit
• Manage Logs Effectively:
  • Forward logs to a centralized logging solution (CloudWatch Logs, Fluentd/Fluent Bit to S3/Elasticsearch).
  • Configure log rotation within containers if logs are written directly to the filesystem.
• Clean Up Container Images: Kubelet automatically garbage collects old images, but if you have many large images or frequent deployments of new images, this can consume significant space.
• Increase Node Disk Size: If ephemeral storage limits are already high and pods legitimately need more space, you might need to use node instance types with larger root volumes or add more nodes. For EKS Managed Node Groups, you would modify the node group configuration to use larger volume sizes.

5. Addressing PID Pressure

PID exhaustion is less common but can be critical.

• Identify High PID Usage Pods: SSH into the node and use tools like htop or ps auxf to identify processes and their PIDs.
• Set PID Limits for Pods: Configure pids limits in your pod definitions. This feature is enabled via the SupportPodPidsLimit feature gate in Kubernetes 1.10+ (and by default in newer versions).
apiVersion: v1 kind: Pod metadata: name: my-high-pid-app spec: containers: - name: my-container image: my-image:latest resources: limits: # Limits the number of PIDs within the container pids: 1024 # Example: adjust based on application needs
• Application Optimization: Review application code for processes that spawn excessive child processes without proper cleanup.
• Node-level PID Limit (Advanced): For EKS, kubelet usually has a default `kube-reserved` for PIDs. Manually changing node-level PID limits for EKS Managed Node Groups is not typically recommended as AWS manages the kubelet configuration. If issues persist, consider scaling up your node group or using larger instance types.

6. Tuning Kubelet Configuration (for EKS Managed Nodes - Use with Caution)

While AWS manages the core kubelet configuration for EKS Managed Node Groups, you can influence some behaviors via Custom Launch Templates associated with your node group or by passing arguments through bootstrap scripts (though less ideal for general kubelet config). Directly modifying kubelet config files on managed nodes is not persistent.

Key eviction thresholds controlled by kubelet:

• --eviction-hard: Hard eviction thresholds (e.g., memory.available<100Mi, nodefs.available<10%, imagefs.available<15%).
• --eviction-soft: Soft eviction thresholds, which include a grace period (e.g., memory.available<200Mi, nodefs.available<15%, imagefs.available<20%).
• --eviction-soft-grace-period: Time before soft eviction is triggered.
• --eviction-max-pod-grace-period: Maximum grace period for pod termination during eviction.
• --system-reserved and --kube-reserved: Specifies reserved resources for system daemons and Kubernetes components, respectively.

To customize these for EKS Managed Node Groups, you would typically define a Custom Launch Template for your node group and use its User data section to pass bootstrap arguments. However, altering core eviction thresholds is a power-user feature and should be done with a deep understanding of its implications. For most cases, optimizing pod resource requests/limits and scaling are preferred solutions.

7. Scaling Strategies

• Horizontal Pod Autoscaler (HPA): Automatically scales the number of pods based on CPU utilization or custom metrics. Prevents a single application from overwhelming its allocated node resources by simply adding more pods across more nodes.
• Cluster Autoscaler (CA): Automatically adjusts the number of nodes in your EKS Managed Node Group's Auto Scaling Group (ASG) based on pending pods and resource requests. This is critical for handling fluctuating workloads and preventing nodes from becoming saturated. Ensure CA is correctly configured for your EKS cluster.
• Vertical Pod Autoscaler (VPA) - Recommendation Only: VPA automatically adjusts resource requests and limits for containers. While it can help right-size pods, it's often used in "recommendation" mode for EKS, as direct VPA control can conflict with HPA.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the likelihood of pod evictions.

1. Define Accurate Resource Requests and Limits

• Always set requests and limits for CPU, memory, and ephemeral storage for all your containers.
• requests determine scheduling, while limits prevent resource hogging.
• Use tools like kube-state-metrics and Prometheus/Grafana to observe actual resource consumption and fine-tune these values.

2. Implement Horizontal Pod Autoscaler (HPA) and Cluster Autoscaler (CA)

• HPA: Scales pods based on actual load, distributing work and preventing resource bottlenecks within a node.
• CA: Scales nodes to accommodate more pods, ensuring your cluster has enough capacity to meet demand. Properly configure the min/max size of your EKS Managed Node Group's underlying ASG.

3. Robust Monitoring and Alerting

• Utilize AWS CloudWatch Container Insights, Prometheus/Grafana, Datadog, or other monitoring solutions.
• Set up alerts for:
  • Node conditions changing to MemoryPressure, DiskPressure, PIDPressure.
  • High CPU, memory, or disk utilization on nodes.
  • Pod eviction events.
  • Low available ephemeral storage.

4. Optimize Container Images and Application Code

• Use minimal base images (e.g., Alpine Linux).
• Implement multi-stage builds to reduce image size.
• Profile applications for memory leaks, inefficient I/O operations, or excessive process spawning.

5. Graceful Shutdown

• Ensure your applications handle SIGTERM signals gracefully, allowing them to clean up resources and finish ongoing requests before termination. This is particularly important during evictions to prevent data loss or inconsistent states.
• Configure terminationGracePeriodSeconds in your pod definition to give applications enough time to shut down.

Frequently Asked Questions (FAQs)

Q1: What is the difference between Pod OOMKill and Node Memory Pressure Eviction?

A1: A Pod OOMKill (Out Of Memory Kill) occurs when a specific container within a pod attempts to use more memory than its configured limits. The Linux kernel's OOM killer terminates that container process to protect the node. A Node Memory Pressure Eviction occurs when the entire node is running low on available memory, regardless of individual pod limits. The kubelet evicts one or more pods from the node to free up resources and restore node stability. OOMKills are per-container; evictions are node-wide and triggered by kubelet.

Q2: Can I disable pod evictions due to node pressure?

A2: While you can technically modify kubelet's eviction thresholds (e.g., by setting --eviction-hard to very high values or disabling certain thresholds), it is strongly discouraged. Evictions are a critical mechanism to maintain node stability. Disabling them will lead to unstable nodes, potential kernel panics, and widespread application failures. The correct approach is to address the root cause of resource pressure, not to suppress the eviction mechanism.

Q3: How do EKS Managed Node Groups affect troubleshooting node pressure?

A3: EKS Managed Node Groups simplify node lifecycle management, including OS patching, updates, and some kubelet configuration. This means direct, persistent modifications to kubelet configuration files on individual nodes are generally not possible or recommended as they can be overwritten. Troubleshooting often focuses on pod resource definitions, application behavior, and cluster-level scaling. For specific node-level adjustments (like reserved resources), you'd typically use Custom Launch Templates with user data to pass bootstrap arguments, or rely on AWS EKS to manage the underlying configuration, using metrics to inform changes to instance types or node group sizes.

By diligently following these guidelines and employing a robust monitoring strategy, you can effectively mitigate and prevent Kubernetes pod evictions due to node pressure on AWS EKS Managed Node Groups, ensuring a resilient and high-performing cloud-native environment.