Deployment and Challenges in Calico's eBPF Mode

Calico is one of the most popular networking solutions for Kubernetes, offering robust support for network policies, IP address management, and efficient routing. In recent versions, Calico introduced support for eBPF (Extended Berkeley Packet Filter), a high-performance data plane that provides significant benefits in terms of speed and flexibility.

However, the use of eBPF mode in Calico introduces a series of challenges, especially when combined with the recommendation to disable kube-proxy in Kubernetes clusters. This article explores these challenges and offers solutions, specifically focusing on ensuring connectivity with the Kubernetes API server in a highly available cluster.

Introduction to Calico eBPF Mode

Calico traditionally used iptables for packet filtering and routing. However, with the introduction of eBPF, Calico now provides a more efficient and flexible networking solution. eBPF operates at a lower level in the Linux kernel, allowing for faster packet processing and the possibility of more complex network operations without the need for iptables.

One key benefit of using Calico with eBPF mode is that it eliminates the need for kube-proxy, which traditionally manages the Service IPs in a Kubernetes cluster. This can lead to performance improvements, particularly in large-scale clusters. However, the official recommendation to disable kube-proxy presents new challenges, especially with the Kubernetes API server connectivity.

Enable the eBPF data plane | Calico Documentation

Step-by-step instructions for enabling the eBPF data plane.

Calico Documentation

Challenges with API Server Connectivity in Calico eBPF Mode

External Load Balancer Requirement

In a standard Kubernetes setup with kube-proxy enabled, services are exposed to the outside world using a combination of iptables rules and IPVS (IP Virtual Server). However, when kube-proxy is disabled in Calico eBPF mode, Kubernetes no longer has a mechanism to route traffic between pods and the API server without an external load balancer.

To ensure reliable communication with the Kubernetes API server, an external load balancer (often implemented using HAProxy or a similar solution) is necessary. This load balancer acts as a proxy for the API server, ensuring that API requests from within the cluster are routed correctly to the appropriate control plane node. The external load balancer is critical because, without it, API traffic may fail to route properly.

DNS Resolution Issues with Kubernetes Service Names

When DNS names are used to connect to the API server within the cluster, Kubernetes uses external DNS service to resolve service names. However, in a Calico eBPF mode setup, the DNS resolution process may cause issues due to the way internal DNS queries are handled. Specifically, clients will attempt to resolve the Kubernetes service names internally, which may lead to lookup errors if the DNS resolution is misconfigured or if there is a delay in resolving the IP addresses of API server endpoints.

This can result in intermittent failures or timeouts when services attempt to communicate with the API server using DNS names. The preferred solution is to avoid relying solely on DNS names and instead use a virtual IP (VIP) for internal communications.

TLS Errors with VIP and Missing SAN Configuration

When using a Virtual IP (VIP) for communication with the Kubernetes API server, one critical issue that may arise is related to the certificate Subject Alternative Name (SAN). If the kubeadm init process did not include the VIP in the API server certificate’s SAN field, communication with the API server will result in TLS handshake failures.

This is a common issue when using VIPs in Kubernetes clusters, particularly in high-availability (HA) setups where API server traffic must be routed through a virtual IP. To avoid this issue, it is essential to ensure that the VIP is added to the SAN during the kubeadm init process. Alternatively, if the VIP was not included, the certificates must be reissued with the correct SAN fields to ensure proper TLS communication.

"Left Foot Meets Right Foot" Problem

A common and particularly challenging issue when setting the KUBERNETES_SERVICE_HOST to the cluster's internal IP (such as 10.96.0.1) is the "left foot meets right foot" problem. This occurs when the control plane is down, and Calico nodes attempt to contact the API server. In this case, the Calico node itself depends on the API server for communication, leading to a circular dependency. Essentially, Calico nodes cannot communicate with the API server because they rely on the cluster IP, which is unreachable due to the failure of the API server.

This creates a scenario where Calico nodes are stuck in a loop, unable to communicate with the API server, and the control plane cannot recover without manual intervention. To resolve this issue, the KUBERNETES_SERVICE_HOST should be set correctly, ensuring that the nodes do not depend on the cluster IP for communication when the control plane is down.

DaemonSet Creation Issues and Dependency Cycles

When Calico nodes are configured to run as DaemonSets, the challenge arises when the API server's Calico node pod is deleted. In such cases, the DaemonSet controller attempts to recreate the pod. However, the DaemonSet itself needs to access the API server to create new pods, which is problematic because the API server, in turn, needs the Calico node to be up for communication. This forms a classic chicken-and-egg problem, where the creation of Calico pods is blocked by the need to communicate with the API server.

This cyclical dependency is another significant issue when the control plane is unavailable, as the system cannot recover automatically. Manual intervention is required to resolve this deadlock, either by restoring the API server manually or adjusting the pod creation logic.

Deployment Steps and Considerations

1. Disable kube-proxy: As recommended for Calico eBPF mode, disable kube-proxy in the Kubernetes configuration. This can be done during the kubeadm init phase or via a configuration change.
2. Use an External Load Balancer: Set up an external load balancer (such as HAProxy) to route traffic to the API server. This load balancer will handle requests from within the cluster and route them to the appropriate control plane node.
3. Ensure VIP Configuration with SAN: During the kubeadm init process, ensure that the VIP is added to the API server certificate SAN. This prevents TLS errors when using the VIP for communication.
4. Use VIP for API Server Connectivity: Configure internal services to connect to the API server using the VIP rather than DNS names. This will avoid DNS lookup errors.
5. Resolve Circular Dependencies: In case of control plane failure, adjust the KUBERNETES_SERVICE_HOST to avoid reliance on the cluster IP. Ensure that the Calico nodes can contact the API server via the VIP even when the control plane is down.
6. Manual Recovery: In cases where the DaemonSet cannot create new Calico pods due to the lack of API server connectivity, manually intervene to restore the control plane and remove any circular dependencies.

Conclusion

Calico eBPF mode offers significant advantages in terms of network performance and flexibility. However, it introduces several challenges, particularly when kube-proxy is disabled and external load balancers are required. Proper planning during the Kubernetes setup phase—such as configuring VIPs with the correct SAN fields and ensuring API server connectivity—is essential to avoid issues like DNS lookup failures and circular dependencies. By addressing these challenges proactively, you can ensure that your Kubernetes cluster remains highly available and performant, even with Calico eBPF mode enabled.