Troubleshooting AWS EKS Pod Network Policy Drops with Calico CNI

Tech Note: Always backup your configuration files before applying any changes to production environments.

Troubleshooting AWS EKS Pod Network Policy Drops with Calico CNI: A Comprehensive Guide

As a Senior Cloud Solution Architect and Software Engineer, I frequently encounter complex networking challenges in Kubernetes environments. One of the most common and often frustrating issues involves unexpected traffic drops within AWS EKS clusters when using Calico as the Container Network Interface (CNI) and network policy engine. This guide provides a comprehensive, step-by-step approach to diagnosing and resolving pod network policy drops, ensuring your applications communicate reliably and securely.

Calico is a powerful and highly scalable network policy engine for Kubernetes, offering advanced features beyond the native Kubernetes NetworkPolicy API. However, its sophisticated nature can sometimes lead to intricate troubleshooting scenarios, especially when policies are misconfigured or when there are underlying issues with the Calico daemon, CNI plugin, or host network stack.

Symptom Analysis & Root Causes

Identifying a network policy drop typically starts with an application experiencing connectivity issues, such as HTTP 5xx errors, connection timeouts, or services being unable to reach dependencies. The key is to differentiate between general network problems and specific policy-related blocks.

Common Symptoms:

• Application Timeouts/Errors: Pods cannot reach other pods, external services, or databases, resulting in connection refused, timeout, or DNS resolution failures.
• Unreachable Services: A service endpoint shows as unreachable from other pods, despite the target pod being healthy.
• Traffic Dumps/Logs: Network diagnostic tools like tcpdump within a pod show traffic being sent but never receiving a response, or firewall logs indicating explicit drops.

Root Causes for Policy Drops:

• Incorrect Network Policy Configuration: This is the most common cause. Policies might be too restrictive, mislabeled selectors, or incorrectly scoped (e.g., using policyTypes: Ingress but needing Egress).
• Policy Order and Precedence: Calico policies are evaluated by order of precedence (GlobalNetworkPolicy > NetworkPolicy). A higher-precedence policy might unintentionally block traffic that a lower-precedence policy intends to allow.
• Missing or Mismatched Labels: Network policies rely heavily on labels. If a pod's labels don't match the policy's podSelector, the policy won't apply or will incorrectly apply.
• Calico Component Health Issues: Problems with calico-node daemon sets, calico-typha, or kube-controllers can prevent policies from being correctly programmed into the host's iptables.
• CNI Plugin Failures: Issues with the Calico CNI plugin itself, such as incorrect configuration or resource exhaustion, can lead to network failures.
• Kernel Module or iptables Conflicts: Rarely, other network configurations or kernel modules on the EKS worker nodes might interfere with Calico's iptables rules.
• IP-in-IP or VXLAN Encapsulation Issues: If your Calico installation uses encapsulation, issues with this layer can manifest as connectivity problems.
• EKS Security Group and Network ACL Conflicts: While Calico handles intra-cluster policy, EKS security groups and VPC Network ACLs still govern traffic at the node and subnet level, especially for external or cross-node communication.

Step-by-Step Resolution Guide

This section provides a structured approach to troubleshoot and resolve network policy drops. Ensure you have kubectl, aws CLI, and calicoctl installed and configured to connect to your EKS cluster.

Prerequisites:

• Kubernetes CLI (kubectl) configured for your EKS cluster.
• AWS CLI (aws) configured for your AWS account.
• Calico CLI (calicoctl) installed.
  
  sudo curl -o /usr/local/bin/calicoctl -L "https://github.com/projectcalico/calico/releases/download/$(curl -s https://api.github.com/repos/projectcalico/calico/releases/latest | grep tag_name | cut -d '"' -f 4)/calicoctl-linux-amd64" sudo chmod +x /usr/local/bin/calicoctl

Step 1: Verify Basic Pod Connectivity

Start by isolating the problem. Can pods communicate at all, or is it specific to certain services?

• Test with a debug pod: Deploy a temporary debug pod (e.g., nicolaka/netshoot) in the same namespace as the problematic pod and try to ping or curl the target pod/service.

kubectl run -it --rm --restart=Never debug-pod --image=nicolaka/netshoot -- /bin/bash # Inside debug-pod ping <target-pod-ip> curl <target-service-name>.<namespace>.svc.cluster.local:<port>

If even basic ping fails, it suggests a deeper network issue beyond just policy.

Step 2: Check Calico Component Status

Ensure all Calico components are running correctly across all worker nodes.

kubectl get pods -n tigera-operator # Or the namespace where Calico is installed (e.g., calico-system) kubectl get ds -n tigera-operator # Check calico-node daemonset

Look for any pods in a Pending, CrashLoopBackOff, or Error state.

kubectl logs -f <calico-node-pod-name> -n tigera-operator kubectl describe pod <calico-node-pod-name> -n tigera-operator

Use calicoctl node status to verify Calico's health on a specific node.

kubectl exec -it <calico-node-pod-name> -n tigera-operator -- calicoctl node status

This output shows the health of Calico on that specific node, including routes and peerings.

Step 3: Analyze Network Policies

Inspect the network policies applied to the affected pods.

• List relevant policies:

kubectl get networkpolicy -n <namespace> calicoctl get networkpolicy -n <namespace> -o yaml calicoctl get globalnetworkpolicy -o yaml

• Examine policy details: Pay close attention to podSelector, policyTypes (Ingress/Egress), and the rules within ingress and egress.

kubectl describe networkpolicy <policy-name> -n <namespace>

• Verify labels: Ensure the labels on your pods match the podSelector of the policy.

kubectl get pod <pod-name> -n <namespace> -o yaml | grep labels

• Policy Precedence: Remember that Calico Network Policies can have an order field (lower number = higher precedence), and GlobalNetworkPolicies always have higher precedence than Kubernetes NetworkPolicies. A higher-order policy might be inadvertently blocking traffic.

Step 4: Debug Policy Enforcement with Calicoctl

calicoctl is invaluable for understanding how policies are being enforced.

• Show policies applied to a workload:

calicoctl get workloadendpoint -n <namespace> <pod-name> -o yaml calicoctl get policy --workload-endpoint=<workload-endpoint-name> -n <namespace>

First, get the workload endpoint name from the workloadendpoint command output (it's often <node-name>-k8s-<pod-name> or similar). Then use that name to list policies.

• Simulate policy enforcement (hypothetical): While Calico doesn't have a direct "dry run" for live policy checks, you can try to craft minimal policies to isolate rules.

Step 5: Inspect Pod iptables Rules (Advanced)

For deep dives, check the actual iptables rules applied by Calico on the host running the pod.

# Find the worker node where the problematic pod is running kubectl get pod <pod-name> -n <namespace> -o wide # SSH into the worker node, then run: sudo iptables-save | grep cali # Or just `sudo iptables-save` and examine output

Calico injects its rules into the FORWARD and INPUT/OUTPUT chains, typically prefixed with cali-. Look for DROP rules that might be blocking your traffic. Be careful interpreting iptables output, it can be complex.

Step 6: Check CNI Configuration

Verify that the CNI plugin on the worker nodes is correctly configured and pointing to Calico.

# SSH into a worker node cat /etc/cni/net.d/10-calico.conflist cat /etc/cni/net.d/calico-kubeconfig.yaml

Also, check the kubelet configuration for the CNI bin directory.

# On worker node ps -aux | grep kubelet | grep cni-bin-dir

Step 7: Review EKS Security Groups and NACLs

While Calico manages pod-to-pod communication within the cluster, AWS security constructs still apply.

• Worker Node Security Groups: Ensure the security group attached to your EKS worker nodes allows necessary ingress/egress for node-to-node communication (if not using IP-in-IP tunnel mode for example) and external access (e.g., to databases outside the cluster).
• VPC Network ACLs: Verify that no restrictive NACLs are blocking traffic between subnets where your EKS nodes reside, especially if your cluster spans multiple subnets.

Use the AWS console or CLI to inspect these resources.

Step 8: Temporarily Relax Policy (Last Resort for Testing)

CAUTION: Only do this in a controlled, non-production environment or with extreme care, as it impacts security.

If you suspect a policy is the issue, you can temporarily relax it or apply a permissive policy to see if connectivity is restored.

• Option 1: Delete specific NetworkPolicies: Temporarily remove the policies you suspect are causing the drop.

kubectl delete networkpolicy <policy-name> -n <namespace>

• Option 2: Apply a "allow all" policy for testing: Create a temporary policy that allows all traffic for the problematic pods.

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-temporary namespace: <your-namespace> spec: podSelector: matchLabels: app: <your-app-label> # Or leave empty {} to select all pods in namespace ingress: - {} egress: - {}

If connectivity is restored after relaxing policies, you've confirmed a policy misconfiguration. Re-introduce policies incrementally or fix the problematic ones.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the incidence of network policy-related issues.

• Policy as Code (GitOps): Manage all network policies in a version-controlled repository (Git) and deploy them via CI/CD pipelines. This ensures consistency, reviewability, and an audit trail.
• Least Privilege Principle: Design policies to allow only the absolutely necessary traffic. Start with restrictive policies and incrementally add rules as needed, rather than starting open and trying to lock down.
• Granular Policies & Clear Labeling: Use precise podSelector and namespaceSelector combinations. Implement a clear, consistent labeling strategy across your Kubernetes resources.
• Regular Audits & Reviews: Periodically review your network policies to ensure they align with current application requirements and security posture. Remove stale or redundant policies.
• Monitoring and Alerting: Implement comprehensive monitoring for Calico components and network traffic. Use tools like Prometheus and Grafana to visualize connection metrics and set up alerts for connectivity failures. Calico provides metrics endpoints that can be scraped.
• Testing Environments: Always test new or modified network policies in a staging or development environment before deploying to production. Consider integration tests that validate network connectivity.
• Calico and EKS Version Compatibility: Ensure your Calico CNI version is compatible with your EKS Kubernetes version. Refer to the official Calico and EKS documentation for compatibility matrices.
• Resource Allocation: Ensure Calico pods (calico-node, calico-typha) have sufficient CPU and memory resources. Resource starvation can lead to delayed policy enforcement or component instability.

Frequently Asked Questions

Q1: What's the difference between Kubernetes NetworkPolicy and Calico NetworkPolicy?

A: Kubernetes NetworkPolicy is a standard API for defining network access rules for pods. Calico implements and extends this API. Calico NetworkPolicies (kind: NetworkPolicy) are more powerful, offering features like policy ordering (via the order field), deny rules, log actions, and support for Host Endpoints, which Kubernetes NetworkPolicy lacks. Calico also provides GlobalNetworkPolicies (kind: GlobalNetworkPolicy) which apply across all namespaces and have higher precedence, making them suitable for cluster-wide security baselines.

Q2: How do I test a network policy without impacting production?

A: The safest way is to test in a dedicated non-production environment that mirrors your production setup. You can also:

1. Deploy a separate "test" namespace with copies of the relevant application components and apply the policy there.
2. Use calicoctl to inspect which policies apply to a test pod before deploying to production.
3. For more advanced scenarios, consider using a network policy "dry run" tool or policy simulator if one is available for Calico, though this is often not a native feature of the CNI itself.

Q3: My policies are correct, but traffic is still dropped. What else could it be?

A: If you're confident your policies are correct and Calico components are healthy, consider these alternative culprits:

• DNS Resolution: Pods might not be able to resolve service names. Check CoreDNS logs and configuration.
• Service Configuration: The Kubernetes Service object itself might be misconfigured (e.g., incorrect port, selector mismatch).
• Kube-proxy Issues: kube-proxy is responsible for service IP tables/IPVS rules. Check its logs and status on worker nodes.
• AWS VPC CNI Interaction: If you're running Calico alongside the AWS VPC CNI (often used for IP address management), ensure there are no conflicts or misconfigurations between them.
• Underlying EC2 Host Networking: Rare but possible, issues with the EC2 instance's network configuration, drivers, or even transient network issues at the AWS infrastructure layer.

Conclusion

Troubleshooting network policy drops in AWS EKS with Calico requires a systematic approach, combining Kubernetes command-line tools with Calico-specific utilities and a solid understanding of network policy mechanics. By following this guide, you can effectively diagnose, resolve, and prevent common connectivity issues, ensuring the robust and secure operation of your containerized applications in the cloud. Remember to prioritize security while maintaining accessibility and always validate changes in controlled environments.