Fixing AWS EKS Pod Network Connectivity Issues with Calico CNI

Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing AWS EKS Pod Network Connectivity Issues with Calico CNI

In the dynamic world of cloud-native applications, AWS Elastic Kubernetes Service (EKS) provides a robust platform for orchestrating containers. However, ensuring seamless network connectivity between pods is paramount for application health and performance. Calico, a popular Container Network Interface (CNI) plugin, extends EKS's networking capabilities, offering advanced network policy enforcement and flexible IP address management. Despite its power, misconfigurations or underlying infrastructure issues can lead to frustrating pod network connectivity problems. This comprehensive guide and troubleshooting manual is designed for Senior Cloud Solution Architects and Software Engineers to diagnose, resolve, and prevent these complex network challenges when using Calico CNI on AWS EKS.

Understanding Calico CNI in AWS EKS

Calico serves as a crucial component in an EKS cluster, primarily responsible for:

• Network Policy Enforcement: Calico implements network policies, allowing fine-grained control over which pods can communicate with each other, enhancing security.
• IP Address Management (IPAM): It provides IP addresses to pods and manages their lifecycle, often using an IP-in-IP or VXLAN overlay network for cross-node communication.
• Routing: Calico ensures that network traffic between pods and to external services is correctly routed across the cluster nodes and out to the internet or other AWS services.

When issues arise, they often stem from a disruption in one of these core functionalities or their interaction with the underlying AWS VPC infrastructure.

Symptom Analysis & Root Causes

Common Symptoms of Pod Network Connectivity Issues

Identifying the symptoms early is key to a swift resolution:

• Pods in CrashLoopBackOff or Error state: Often due to failed liveness/readiness probes that depend on network access.
• Connection refused or Connection timeout errors: When applications within pods try to communicate with other pods, services, or external endpoints.
• DNS resolution failures: Pods unable to resolve internal Kubernetes service names or external hostnames.
• High latency or packet loss: Degraded network performance impacting application responsiveness.
• kubectl logs showing network-related errors: Application logs explicitly indicating connectivity problems.
• ping or curl from within a pod fails: Basic network diagnostic tools not working as expected.

Potential Root Causes

The complexity of Kubernetes networking means issues can arise from multiple layers:

• Calico DaemonSet Issues: calico-node pods not running, stuck in Pending, or repeatedly crashing due to resource constraints, incorrect configuration, or underlying node problems.
• Misconfigured Calico NetworkPolicy: Overly restrictive policies blocking legitimate traffic between pods or to services.
• AWS Security Group (SG) and Network ACL (NACL) Rules: Inadequate inbound/outbound rules on worker node security groups or VPC NACLs preventing inter-node pod communication or control plane communication.
• IP Exhaustion or Calico IP Pool Issues: Not enough IP addresses available in the Calico IP pool, or misconfiguration of the IPAM.
• kubelet or containerd Errors: Issues with the Kubelet not properly integrating with the CNI plugin or the container runtime failing to set up the pod's network namespace.
• IPTables Conflicts or Corruption: Calico heavily relies on iptables for routing and network policy enforcement. Conflicts with other software or kernel issues can corrupt rules.
• MTU Mismatches: Inconsistent Maximum Transmission Unit (MTU) settings across the network path (EKS nodes, VPC, VPN/Direct Connect) can lead to packet fragmentation and loss.
• Kernel Parameters: Incorrect sysctl settings on worker nodes affecting network behavior.

Step-by-Step Resolution Guide

Prerequisites

Before you begin, ensure you have:

• kubectl configured to access your EKS cluster.
• aws CLI installed and configured for your AWS account.
• ssh access to your EKS worker nodes.
• Familiarity with Calico concepts and Kubernetes networking.

Step 1: Verify Calico Pod Status and Logs

Start by ensuring that all Calico components are running correctly across your cluster. Issues with Calico pods are often the root cause of network problems.

kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get pods -n kube-system -l k8s-app=calico-kube-controllers

Look for all pods to be in a Running state. If any are Pending, Error, or CrashLoopBackOff, investigate further:

kubectl describe pod <calico-node-pod-name> -n kube-system
kubectl logs <calico-node-pod-name> -n kube-system --tail=100

Pay attention to events, error messages, and restart counts. Common issues include resource limits, image pull errors, or configuration problems.

Step 2: Check NetworkPolicy Conflicts

Calico NetworkPolicies can be highly restrictive. Even a single misconfigured policy can block critical traffic. Identify and review existing policies.

kubectl get networkpolicy --all-namespaces
kubectl get globalnetworkpolicy

For a suspicious policy, inspect its details:

kubectl describe networkpolicy <policy-name> -n <namespace>

Troubleshooting Tip: As a diagnostic step, you can temporarily create a very permissive NetworkPolicy in the affected namespace or for the affected pod to see if connectivity resumes. If it does, your issue is definitely policy-related.

Step 3: Inspect AWS Security Groups and Network ACLs

AWS network constructs underpin your EKS cluster. Ensure your worker node Security Groups (SGs) and VPC Network ACLs (NACLs) allow necessary traffic.

• Worker Node SGs:
  • Allow all traffic (TCP, UDP, ICMP) between worker nodes on all ports (for Calico's overlay or direct routing). This is typically achieved by allowing traffic from the worker node SG itself.
  • Allow inbound from Control Plane SG (e.g., TCP 443, 10250-10252, etc.).
  • Allow outbound to Control Plane SG (e.g., TCP 443).
• VPC NACLs: Ensure NACLs associated with subnets where worker nodes reside are permissive enough (e.g., allow all inbound/outbound ephemeral ports, and specific ports for services if not completely open).

Use the AWS Management Console or AWS CLI to inspect these:

aws ec2 describe-security-groups --filters Name=tag:eks:cluster-name,Values=<your-cluster-name> --query 'SecurityGroups[*].[GroupName,GroupId,IpPermissions,IpPermissionsEgress]' --output json

Step 4: Validate Calico Configuration (IPAM, IP Pool)

Calico's IPAM relies on IPPools. Ensure they are correctly defined and not exhausted.

kubectl get ippool -o yaml

Check the cidr range, blockSize, and importantly, the ipipMode or vxlanMode. If ipipMode: Never or vxlanMode: Never is set, ensure your underlying AWS VPC routing can handle direct pod-to-pod routing, which is less common for cross-node communication in EKS without the AWS VPC CNI. Calico usually requires IP-in-IP or VXLAN for cross-node connectivity.

Also, inspect the main Calico ConfigMap for any misconfigurations:

kubectl get configmap -n kube-system calico-config -o yaml

Look for parameters like CALICO_IPV4POOL_CIDR, CALICO_IPV4POOL_IPIP, CALICO_IPV4POOL_VXLAN, and CALICO_AWS_PREFIX.

Step 5: Review kubelet and containerd Logs

If Calico pods are healthy but application pods still have issues, the problem might be closer to the container runtime or Kubelet. SSH into an affected worker node:

ssh ec2-user@<worker-node-private-ip>

Then check the Kubelet and container runtime logs:

sudo journalctl -u kubelet -l --no-pager | grep -i "cni\|network\|error" --since "10 minutes ago"
sudo journalctl -u containerd -l --no-pager | grep -i "cni\|network\|error" --since "10 minutes ago"

Look for errors related to CNI plugin execution, network interface setup, or IP address assignment. Also, check for kernel messages:

dmesg | tail -n 50

Step 6: Check for IPTables Issues

Calico manages an extensive set of iptables rules. Incorrect or conflicting rules can severely disrupt traffic. From an affected worker node, inspect the iptables chains managed by Calico:

sudo iptables-save | grep -E "cali-|KUBE-FWD"

Verify the rules look sensible for your configuration. Also, check the routing table to ensure the pod CIDR ranges are correctly routed:

ip route show

And confirm that tunnels (IP-in-IP or VXLAN) are up and routing traffic if configured:

ip tunnel show
ip -d link show caliX # Replace caliX with your tunnel interface, e.g., tunl0 or vxlan.calico

Step 7: Reinstall/Upgrade Calico CNI (As a last resort)

If all other steps fail, a full reinstallation or upgrade of Calico might resolve persistent issues, especially after major EKS upgrades or if Calico components are corrupted. Ensure you use a Calico version compatible with your EKS version.

Note: This will temporarily disrupt networking for all pods using Calico. Plan this during a maintenance window.

# First, find the Calico manifest URL for your desired version # e.g., for EKS 1.28 and Calico 3.26: # CALICO_MANIFEST_URL="https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml" # Delete existing Calico deployment kubectl delete -f ${CALICO_MANIFEST_URL} # Wait for resources to be terminated sleep 30 # Reapply Calico kubectl apply -f ${CALICO_MANIFEST_URL}

Monitor the calico-node and calico-kube-controllers pods carefully after reinstallation.

Best Practices for Prevention & Performance Optimization

Proactive measures can significantly reduce the likelihood of network connectivity issues:

• NetworkPolicy Management:
  • Implement policies using a "deny-by-default, allow-by-exception" approach for enhanced security, but test thoroughly.
  • Use network policy tools (e.g., Calico Policy Reporter, Kubevious) for visualization and auditing.
  • Regularly review and prune stale or overly broad policies.
• Monitoring and Alerting:
  • Monitor Calico pod health (calico-node, calico-kube-controllers) with Prometheus/Grafana or AWS CloudWatch.
  • Set up alerts for high error rates in Calico pod logs or unreachable endpoints.
  • Monitor EKS worker node network metrics (packet drops, latency).
• AWS Network Configuration:
  • Ensure Security Groups and NACLs are correctly configured and follow the principle of least privilege, but without blocking essential EKS/Calico traffic.
  • Validate VPC route tables for proper inter-subnet and internet routing.
• IPAM Planning:
  • Plan your Calico IPPool CIDR ranges carefully to avoid exhaustion and overlap with other VPC subnets.
  • Consider using larger blockSize for IPPools if you have a high churn of pods.
• Version Compatibility:
  • Keep your EKS and Calico CNI versions up-to-date and ensure compatibility between them.
  • Test upgrades in a staging environment before applying to production.
• MTU Consistency: Ensure consistent MTU settings across all EKS nodes and VPC network interfaces to prevent fragmentation. For IP-in-IP or VXLAN, the effective MTU for pods will be lower than the host's MTU.

Frequently Asked Questions

Q1: How do I verify Calico is correctly installed and functioning on EKS?

A1: Beyond checking pod statuses (Step 1), you can use calicoctl for a more in-depth look. First, install calicoctl. Then, run calicoctl node status on a worker node to see its Calico health and peering status. You can also use calicoctl get ippools to verify your IP address management configuration.

# Example for installing calicoctl (adjust version as needed) # curl -L https://github.com/projectcalico/calico/releases/download/v3.26.1/calicoctl-linux-amd64 -o /usr/local/bin/calicoctl # chmod +x /usr/local/bin/calicoctl calicoctl node status

Q2: What's the fundamental difference between Calico and Amazon VPC CNI in EKS, and why choose Calico?

A2: The Amazon VPC CNI assigns native VPC IP addresses to pods, allowing them to directly participate in the VPC network. It's often simpler for basic networking. Calico, on the other hand, typically uses an overlay network (IP-in-IP or VXLAN) or direct routing with BGP, assigning IPs from its own defined IP pools. The primary reason to choose Calico is its superior, highly granular Network Policy capabilities, allowing for complex security rules and integration with advanced features like Global Network Policies and Host Endpoints, which VPC CNI does not natively provide.

Q3: My pods are stuck in ContainerCreating, and Kubelet logs mention CNI plugin errors. Is this a Calico issue?

A3: Yes, this is a strong indicator of a CNI-related issue, which could very likely be Calico. When a pod gets stuck in ContainerCreating, Kubelet is often waiting for the CNI plugin (Calico in this case) to configure the pod's network namespace and assign an IP address. Check the kubelet and calico-node logs on the affected worker node (as detailed in Step 5 and 1) for specific errors. Common causes include Calico daemonset not running, IP pool exhaustion, or permission issues preventing Calico from interacting with the host's network stack (iptables, routes).