Tech Note: Always backup your configuration files before applying any changes to production environments.

Debugging EKS Pod Readiness Probe Failures with Custom Health Checks

In the dynamic world of cloud-native applications on Amazon Elastic Kubernetes Service (EKS), ensuring the reliability and availability of your services is paramount. Pod readiness probes are a critical component of Kubernetes that dictate whether a pod is ready to serve traffic. When these probes fail, it can lead to service disruptions, incomplete deployments, and a degraded user experience. This comprehensive guide, authored by a Senior Cloud Solution Architect and Software Engineer, delves into the intricacies of debugging EKS Pod readiness probe failures, with a particular focus on scenarios involving custom health checks.

Understanding Pod Readiness Probes in EKS

Kubernetes uses readiness probes to determine when a container is ready to start accepting traffic. A pod is considered ready when all its containers are ready. If a readiness probe fails, Kubernetes removes the pod's IP address from the endpoints of all services that match the pod. This prevents traffic from being routed to an unhealthy or unready pod. Custom health checks, often implemented via exec probes or specific HTTP endpoints, offer greater flexibility in defining what "ready" truly means for complex applications.

Symptom Analysis & Root Causes

Identifying the symptoms and understanding the root causes of readiness probe failures is the first step toward effective debugging.

Common Symptoms:

• Pods stuck in NotReady state: kubectl get pods will show one or more containers in a pod as 0/1, 1/2, etc., under the READY column.
• Deployment failures or prolonged rollouts: New pods never become ready, causing deployments to hang or revert.
• Service unavailability or intermittent errors: Even if pods are running, if they aren't ready, load balancers won't direct traffic to them, leading to 5xx errors or timeouts for users.
• Repeated pod restarts: While more common with liveness probes, a readiness probe misconfiguration might indirectly lead to restart loops if other systems are tightly coupled.

Potential Root Causes:

• Application Not Listening: The application inside the container is not running, not listening on the expected port/path, or is still initializing.
• Incorrect Probe Configuration:
  • Wrong port or path: The probe tries to connect to a port or HTTP path where the application is not exposing its health endpoint.
  • Incorrect exec command: The custom script/command specified in the exec probe does not exist, lacks execution permissions, or returns a non-zero exit code for success.
  • Insufficient initialDelaySeconds: The application takes longer to start than the specified delay, causing the probe to fail prematurely.
  • periodSeconds too short: The probe checks too frequently, overwhelming a slow-starting or resource-constrained application.
  • timeoutSeconds too low: The probe times out before the application can respond, especially for complex health checks.
• Resource Exhaustion:
  • CPU/Memory limits: The container is starved of resources, causing the application or health check process to become unresponsive.
  • Disk I/O contention: Heavy disk operations during startup delay readiness.
• Network Issues:
  • Security Group/Network ACL misconfiguration: Blocking traffic between the kubelet (on the node) and the pod's health endpoint.
  • CNI plugin issues: Underlying network fabric problems preventing connectivity.
• Application Logic Errors:
  • Deadlocks, infinite loops, or unhandled exceptions within the application preventing it from reaching a "ready" state.
  • External dependency failures (database, message queue, external API) that the custom health check is designed to monitor.
• Custom Health Check Specific Problems:
  • The script itself has bugs, takes too long to execute, or performs checks that are too stringent for initial readiness.
  • Dependencies for the custom script are missing inside the container (e.g., curl, jq, custom binaries).

Step-by-Step Resolution Guide for Readiness Probe Failures

This systematic approach will guide you through diagnosing and fixing readiness probe issues on EKS.

Step 1: Verify Pod Status and Events

Start by inspecting the pod's current state and recent events. This often provides immediate clues.

kubectl get pods -n <namespace> kubectl describe pod <pod-name> -n <namespace>

Look for: The Events section in kubectl describe will often show specific probe failures (e.g., Readiness probe failed: HTTP GET http://... connection refused or Readiness probe failed: exit status 1).

Step 2: Inspect Pod Logs

Application logs are crucial for understanding what's happening inside the container during startup and probe execution.

kubectl logs <pod-name> -n <namespace> --previous kubectl logs <pod-name> -n <namespace> -c <container-name>

Look for: Application startup errors, port binding issues, unhandled exceptions, or specific messages related to your custom health check logic.

Step 3: Check Probe Configuration in Pod YAML

Review the readinessProbe section in your deployment or pod definition. Ensure that all parameters are correctly set.

kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A 10 "readinessProbe"

Key parameters to check:

• initialDelaySeconds: Is it long enough for your application to fully start and initialize?
• periodSeconds: How often is the probe executed? Too frequent can sometimes stress the application.
• timeoutSeconds: Does the probe have enough time to get a response?
• failureThreshold: How many consecutive failures before the pod is marked NotReady?
• For httpGet: port, path, scheme. Ensure they match your application's health endpoint.
• For tcpSocket: port. Ensure the application is truly listening on this port.
• For exec: The command array. Is the command syntactically correct, does it exist inside the container, and does it return a 0 exit code on success?

Step 4: Manually Test the Health Check Inside the Pod

This is a critical step for custom health checks. Execute the probe's command or access the endpoint directly from within the problematic pod.

# Get a shell into the problematic container kubectl exec -it <pod-name> -n <namespace> -c <container-name> -- /bin/bash # or /bin/sh # If httpGet probe (replace with your port and path) curl -v http://localhost:<port>/<path> # If tcpSocket probe (you might need to install netcat 'apt update && apt install -y netcat') nc -vz localhost <port> # If exec probe (execute the exact command from your YAML) /usr/bin/healthcheck.sh echo $? # Check the exit code (0 for success, non-zero for failure)

Analyze the output:

• curl: Look for HTTP status codes (2xx for success), connection refused errors, or timeouts.
• nc: Confirm if the connection is successful.
• exec command: Does it run without errors? What is the exit code? If it's a script, add set -x at the top to trace execution, or add echo statements for debugging. Check if any required binaries are missing.

Step 5: Resource Analysis

Resource constraints can cause applications to become unresponsive, leading to probe failures.

kubectl top pod <pod-name> -n <namespace> --containers kubectl describe node <node-name-where-pod-is-running>

Look for: High CPU/Memory usage close to or exceeding configured limits. Check for OOMKilled events. If the node itself is overloaded, it might impact pod performance.

Step 6: Network Troubleshooting

Ensure network connectivity between the kubelet (on the node) and the pod's health endpoint.

• Security Groups/Network ACLs: On AWS EKS, verify that the Security Group attached to your EKS nodes allows outbound traffic to your pods (often handled by the default EKS setup, but custom configurations can break this). Similarly, ensure your pod's security groups allow inbound traffic from the node.
• CNI Plugin Logs: If using AWS VPC CNI, check the logs of the aws-node pods for network-related errors.
• Network Policies: If Kubernetes Network Policies are in use, ensure they allow the kubelet to reach the pod's health check port.

Step 7: Refine Custom Health Check Logic (for exec probes)

If your custom script is failing, focus on its internal logic:

• Exit Codes: Remember that Kubernetes expects an exit code of 0 for success and any non-zero for failure.
• Permissions: Ensure the script has execute permissions (e.g., chmod +x /path/to/script.sh within your Dockerfile).
• Dependencies: Confirm all binaries (curl, jq, etc.) and environment variables needed by the script are present in the container image.
• Idempotence & Speed: Health checks should be fast and idempotent. Avoid heavy computations or operations that modify state.

# Example of a robust custom health check script (healthcheck.sh) #!/bin/bash set -eo pipefail # Exit immediately if a command exits with a non-zero status # Check if the application's main process is running pgrep my_app_process > /dev/null || exit 1 # Check an internal HTTP endpoint curl -sS http://localhost:8080/health > /dev/null || exit 1 # Check database connectivity (example) # psql -h <db_host> -U <db_user> -c "SELECT 1" &> /dev/null || exit 1 echo "Application is healthy." exit 0 # Success

Step 8: Temporarily Adjust Probe Parameters

As a diagnostic step, temporarily increase initialDelaySeconds, periodSeconds, or timeoutSeconds to see if the probe eventually succeeds. This can help pinpoint if the issue is timing-related.

# Example K8s YAML snippet readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 60 # Increased from default 10-15s periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3

Step 9: Review External Dependencies

If your health check involves external services (databases, message queues, external APIs), ensure those services are available and reachable from your EKS cluster.

• Check DNS resolution from within the pod.
• Verify network connectivity (ping, telnet, nc) to external endpoints.
• Inspect logs of those external services.

Step 10: Rollback or Reiterate

If a recent change introduced the issue, consider rolling back to a known good configuration. If the problem persists, iterate through these debugging steps, applying minor changes and re-testing until the root cause is identified.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce readiness probe failures.

Design Robust Health Checks:

• Differentiate Liveness and Readiness:
  • Liveness Probe: Should only check if the application is still running. If it fails, Kubernetes restarts the container. A shallow check (e.g., a simple HTTP 200) is often sufficient.
  • Readiness Probe: Should check if the application is ready to serve traffic. This can include checking database connections, message queues, or other critical dependencies. This might be a deeper check.
• Graceful Shutdown: Implement graceful shutdown in your applications, giving them time to finish processing requests before terminating. Combine this with preStop hooks.
• Configuration via Environment Variables: Allow health check parameters (ports, paths, delays) to be configurable via environment variables, enabling easier adjustments without rebuilding images.

Optimize Application Startup:

• Minimize Image Size: Smaller images download faster, leading to quicker pod startups. Use multi-stage builds and Alpine base images where possible.
• Lazy Loading: Initialize non-critical components after the initial readiness state is achieved.

Effective Resource Management:

• Set Appropriate Requests and Limits: Configure requests (guaranteed resources) and limits (maximum resources) for CPU and memory. Monitor application usage to set these accurately.
• Horizontal Pod Autoscaling (HPA): Use HPA to scale your application based on resource usage or custom metrics, preventing resource exhaustion during peak loads.

Robust Observability:

• Centralized Logging: Forward container logs to a centralized logging solution (e.g., CloudWatch Logs, Splunk, ELK stack).
• Monitoring and Alerting: Monitor Kubernetes events, pod states, and application metrics. Set up alerts for frequent readiness probe failures.
• Distributed Tracing: For microservices, implement distributed tracing to understand request flow and identify bottlenecks, which can indirectly affect readiness.

Staging & Testing:

• Always test your readiness probes thoroughly in non-production environments. Simulate various failure scenarios (e.g., database downtime, network latency) to see how your probes react.

Frequently Asked Questions

Q1: What is the fundamental difference between Liveness and Readiness probes?

A1: Liveness probes determine if a container is running correctly. If a liveness probe fails, Kubernetes restarts the container. This is for recovery from application deadlocks or unrecoverable states. Readiness probes determine if a container is ready to serve traffic. If a readiness probe fails, Kubernetes removes the pod's IP from the Service endpoints, preventing traffic from being sent to it. This allows applications to warm up or temporarily halt traffic during maintenance without being restarted.

Q2: My application takes a long time to start. How do I prevent readiness probe failures during startup?

A2: The key is to correctly configure initialDelaySeconds in your readiness probe. This parameter tells Kubernetes to wait for a specified duration before starting the first probe check. Set this value to be slightly longer than your application's typical startup time. You can also increase periodSeconds to check less frequently, giving the application more time between checks, and failureThreshold to allow for a few transient failures.

Q3: The exec probe always fails, but the command works when I run it manually inside the pod. What could be wrong?

A3: This is a common issue with exec probes. Here are potential culprits:

• Permissions: Ensure the script has execute permissions (chmod +x).
• Absolute Paths: Always use absolute paths for commands and scripts (e.g., /usr/bin/curl instead of curl, /app/healthcheck.sh instead of healthcheck.sh). The environment PATH might be different for the probe.
• Shell vs. Command: If your command is complex or uses shell features (like pipes, redirects), wrap it in a shell invocation (e.g., command: ["/bin/sh", "-c", "curl -sS http://localhost:8080/health || exit 1"]).
• Environment Variables: The probe environment might not have all the environment variables that are present when you manually exec into the pod. Explicitly pass any required variables to your script.
• Exit Code: Double-check that your script explicitly exits with 0 for success and a non-zero value for failure.

Conclusion

Debugging EKS Pod readiness probe failures, especially with custom health checks, requires a systematic approach and a deep understanding of both Kubernetes and your application's behavior. By following the troubleshooting steps outlined in this guide and adopting best practices for prevention, you can ensure the high availability and reliability of your cloud-native applications on Amazon EKS. Remember, effective observability through logging, monitoring, and tracing is your most powerful ally in maintaining a robust and resilient EKS environment.