Troubleshooting Kubernetes Liveness Probe Failure in AWS EKS Due to High CPU Throttling

Tech Note: Always backup your configuration files before applying any changes to production environments.

Introduction: Understanding Liveness Probe Failures in AWS EKS

Kubernetes Liveness probes are essential for maintaining the health and availability of applications deployed on clusters like AWS EKS. They inform the Kubelet when a container is unresponsive and needs to be restarted. While incredibly powerful, misconfigurations or underlying resource constraints can lead to false positives, causing unnecessary restarts and service disruptions. One common yet often overlooked culprit for persistent Liveness probe failures is high CPU throttling, where a container's CPU usage exceeds its defined limits, leading to performance degradation and an unresponsive application, even if the application itself isn't crashed.

This comprehensive guide provides a deep dive into diagnosing and resolving Liveness probe failures in AWS EKS environments specifically caused by CPU throttling. We'll cover symptom analysis, root causes, a step-by-step resolution, and best practices to prevent these issues.

Symptom Analysis & Root Causes

Understanding the symptoms and their underlying causes is the first step towards effective troubleshooting. CPU throttling can manifest in various ways, often mimicking other application issues.

Symptoms of CPU Throttling Causing Liveness Probe Failure:

• Frequent Pod Restarts: The most apparent symptom is a pod repeatedly entering a CrashLoopBackOff or Evicted state due to Liveness probe failures.
• Liveness Probe Failure Logs: Kubelet events or pod logs will show messages indicating that the Liveness probe failed, often with timeouts. E.g., Liveness probe failed: Get "http://localhost:8080/health": context deadline exceeded.
• Application Unresponsiveness: Even when not restarting, the application within the pod may become slow, unresponsive to requests, or exhibit high latency, especially under load.
• High CPU Utilization Metrics: Monitoring tools (Prometheus, CloudWatch Container Insights) will show high CPU usage for the affected container, often hitting or exceeding its configured CPU limit.
• Increased ThrottledPeriods and ThrottledTime: Specific container metrics (e.g., from cAdvisor, integrated into Prometheus) will indicate a significant number of CPU throttled periods and accumulated throttled time.
• No OOMKills (Out Of Memory Kills): Unlike memory issues, CPU throttling typically doesn't result in OOMKills, but rather a slowdown or freeze of the process.

Root Causes of High CPU Throttling:

• Inadequate CPU Limits: The most common cause. The limits.cpu setting in the pod's container definition is set too low for the application's actual workload, leading to the kernel aggressively throttling the container's CPU access when it exceeds the limit.
• Bursty Workloads: Applications that have intermittent spikes in CPU usage may hit their limits frequently, even if their average CPU usage is low.
• CPU-Bound Application: The application itself is inherently CPU-intensive, and the allocated CPU resources are simply insufficient for its operational demands.
• Inefficient Application Code: Poorly optimized code can consume more CPU than necessary, leading to throttling even with seemingly adequate resource allocations.
• Overly Aggressive Liveness Probe Configuration: While not a direct cause of throttling, a Liveness probe with a very short periodSeconds or timeoutSeconds can fail quickly when the application experiences even minor throttling-induced delays, triggering restarts.
• Node Resource Exhaustion: In some cases, the underlying EKS worker node might be generally overloaded, leading to resource contention and making it harder for pods to get their requested CPU share, even if their individual limits are reasonable.

Step-by-Step Resolution Guide

Follow these steps to diagnose and resolve Liveness probe failures caused by CPU throttling in your AWS EKS environment.

Preparation: Ensure you have necessary tools and access.

Before proceeding, ensure you have kubectl configured with access to your EKS cluster and necessary permissions to view and modify deployments.

Step 1: Identify and Confirm CPU Throttling

First, verify that CPU throttling is indeed the root cause. This involves checking pod metrics and events.

• Check Pod CPU Usage: Use kubectl top pod to get a quick overview of CPU usage for pods. Look for pods consistently using near their CPU limit.

kubectl top pod -n <your-namespace>

• Examine Pod Events and Status: Use kubectl describe pod to check for Liveness probe failures and related events. Look for clues like Liveness probe failed in the Events section.

kubectl describe pod <pod-name> -n <your-namespace>

• Check for Throttled Metrics: If you have Prometheus and Grafana or AWS CloudWatch Container Insights, examine the container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total metrics for the specific container. A sustained increase in these metrics confirms throttling.

Step 2: Analyze Current CPU Limits and Requests

Once throttling is confirmed, inspect the current CPU resource definitions for the affected container.

• Get Pod YAML: Retrieve the YAML definition of the failing pod to identify its configured CPU requests and limits.

kubectl get pod <pod-name> -n <your-namespace> -o yaml

Look for the resources section within your container definition:

resources: limits: cpu: "500m" # 0.5 CPU core memory: "512Mi" requests: cpu: "250m" # 0.25 CPU core memory: "256Mi"

Note: m stands for millicores, where 1000m equals 1 CPU core.

Step 3: Adjust CPU Resources (Requests and Limits)

Based on your findings, increase the CPU limits (and often requests) for the affected container. It's crucial to adjust these in your deployment manifest (e.g., deployment.yaml), not directly on the pod, as direct pod changes are often overwritten.

• Modify Your Deployment/StatefulSet/DaemonSet YAML: Edit the .spec.template.spec.containers[].resources.limits.cpu and .spec.template.spec.containers[].resources.requests.cpu values. A common strategy is to start by setting limits.cpu higher (e.g., double the current value) and observe the effect. For requests.cpu, it's often a good practice to set it equal to limits.cpu for critical applications to prevent CPU starvation and ensure Quality of Service (QoS) guarantees, or slightly lower if you want to allow for bursting beyond the request but still within a limit.

# Example deployment.yaml snippet apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: template: spec: containers: - name: my-container image: my-image:latest resources: limits: cpu: "1000m" # Increased from 500m to 1000m (1 full CPU core) memory: "1Gi" requests: cpu: "500m" # Increased from 250m to 500m memory: "512Mi" livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 30 # Consider increasing this periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3

• Apply the Changes: Once modified, apply your YAML file to the cluster. This will trigger a rolling update, recreating your pods with the new resource settings.

kubectl apply -f your-deployment.yaml -n <your-namespace>

Step 4: Optimize Liveness Probe Configuration

While increasing CPU resources is primary, adjusting the Liveness probe parameters can make it more resilient to temporary slowdowns.

• initialDelaySeconds: Increase this to give the application enough time to fully start up and warm up without prematurely failing the probe.
• periodSeconds: Lengthen the interval between probe checks. A longer period gives the application more time to recover from a momentary resource constraint before the next check.
• timeoutSeconds: Increase the time Kubelet waits for a response from the probe. This is crucial for CPU throttling scenarios, as a throttled application might respond slowly but is not truly "down."
• failureThreshold: Increase the number of consecutive failures required before Kubelet considers the pod unhealthy and restarts it. This allows for transient issues without immediate restarts.

# Example Liveness Probe configuration with optimized values livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 60 # Give 60 seconds for startup periodSeconds: 15 # Check every 15 seconds timeoutSeconds: 10 # Wait up to 10 seconds for a response failureThreshold: 5 # Allow 5 consecutive failures before restart

Step 5: Monitor and Validate

After applying changes, rigorously monitor your pods and application performance.

• Check Pod Status: Continuously monitor kubectl get pods -n <your-namespace> -o wide to ensure pods remain in a Running state and don't re-enter CrashLoopBackOff.
• Review Logs and Events: Use kubectl logs <pod-name> and kubectl describe pod <pod-name> to check for Liveness probe failures or other errors.
• Performance Monitoring: Utilize your monitoring stack (Prometheus, Grafana, CloudWatch) to observe CPU usage, request latency, and the absence of CPU throttling metrics (container_cpu_cfs_throttled_periods_total should be flat or very low).
• Load Testing: If possible, subject the application to load to verify stability under expected traffic conditions.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the likelihood of CPU throttling and Liveness probe failures.

• Accurate Resource Requests and Limits:
  • Start with Requests < Limits: Set requests.cpu to the average expected usage and limits.cpu to the maximum acceptable usage. This allows for some bursting.
  • Monitor and Adjust: Continuously monitor actual CPU usage and adjust requests/limits based on real-world performance data.
  • Equal Requests & Limits for Critical Apps: For critical, performance-sensitive applications, setting requests.cpu = limits.cpu ensures a Guaranteed QoS class, preventing CPU starvation.
• Optimize Liveness and Readiness Probes:
  • Use startupProbe: For applications with long startup times, use a startupProbe to delay Liveness probe checks until the application is fully initialized.
  • Sensible Parameters: Configure initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold to be forgiving enough for temporary slowdowns but strict enough to detect actual failures.
  • Lightweight Probes: Ensure your probe endpoints are lightweight and do not consume significant resources themselves.
• Horizontal Pod Autoscaler (HPA): Implement HPA based on CPU utilization to automatically scale out (add more pods) when average CPU usage exceeds a defined threshold, distributing the load and preventing individual pods from throttling.
• Vertical Pod Autoscaler (VPA): Consider VPA (if compatible with HPA in your setup) to automatically adjust CPU and memory requests/limits for your pods over time, learning from historical usage patterns.
• Application Performance Monitoring (APM) & Profiling:
  • Identify Bottlenecks: Use APM tools to profile your application code and identify CPU-intensive sections that can be optimized.
  • Right-Size Containers: Understand your application's resource demands through profiling and monitoring to allocate resources more accurately.
• EKS Node Group Sizing: Ensure your underlying EKS worker nodes (EC2 instances) are appropriately sized and have sufficient available CPU capacity to accommodate the pods you schedule. Over-provisioning nodes slightly can provide a buffer.

Frequently Asked Questions (FAQs)

1. What is the difference between CPU 'requests' and 'limits' in Kubernetes?

CPU Request: This is the amount of CPU guaranteed to the container. The Kubernetes scheduler uses this value to decide which node to place the pod on. The node must have at least this much CPU capacity available. If a container requests 0.5 CPU, it is guaranteed at least 0.5 CPU.
CPU Limit: This is the maximum amount of CPU the container is allowed to use. If a container tries to use more CPU than its limit, it will be throttled. This means its execution will be paused until its CPU usage falls below the limit, preventing it from consuming all CPU resources on a node and affecting other pods. Setting limits too low is the primary cause of CPU throttling.

2. How can I definitively check if my pod is experiencing CPU throttling?

The most definitive way is to examine container metrics, specifically container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total. These metrics, typically exposed by cAdvisor (which Kubelet integrates with) and collected by monitoring systems like Prometheus, directly report when and for how long a container's CPU usage has been throttled by the Linux kernel's CFS (Completely Fair Scheduler). A non-zero or increasing value in these metrics confirms throttling.

3. Should I use startupProbe in conjunction with livenessProbe?

Yes, absolutely. For applications with non-trivial startup times, it's highly recommended to use a startupProbe. The startupProbe delays the execution of both the readinessProbe and livenessProbe until the application successfully signals it has started. This prevents premature Liveness probe failures (and restarts) during the application's initial warm-up phase, which can be particularly CPU-intensive. Once the startupProbe succeeds, the regular livenessProbe takes over.