Making Kubernetes the Network Control Plane: Public Service IPs with Cilium, BGP and ExternalDNS

Previously, Kubernetes could report a private LoadBalancer address while the actual public endpoint belonged to a separate network appliance. Exposing an application meant keeping the Service, the appliance configuration, and DNS consistent. Kubernetes described the application, but it did not describe its complete public identity.

The target is simpler: the LoadBalancer address is the public address. Cilium allocates it, advertises a route toward the appropriate nodes, and implements Service forwarding through eBPF. That same address can become the DNS target without an engineer copying it into another system.

The cluster now runs without kube-proxy. ExternalDNS and Google Cloud DNS provide the integration pattern that completes this cluster-centered model; the DNS configuration below describes that integration, rather than claiming an already completed production DNS rollout.

About the examples: This article reconstructs the environment rather than publishing the production topology. Addresses, ASNs, hostnames, node counts, and prefix sizes are synthetic. Internal networks use 10.0.0.0/8; public-looking examples use 20.72.0.0/16. The latter is not documentation-reserved address space, and no ownership of it is implied. Do not deploy or advertise these example public addresses. The three migration incidents are drawn from execution notes, with identifying details removed. IANA address-space registry^[1].

The Service IP Was Not the Public Endpoint

In Analyzing Load Balancer VIP Routing with Calico BGP and MetalLB^[2], I examined a stack in which MetalLB allocated Service addresses, Calico advertised them, and the node's Service datapath handled delivery to backends. Address allocation, routing, and forwarding were separate responsibilities.

The environment behind this migration had an additional separation: the public ingress path used an edge appliance forwarding to worker hostPorts. A private LoadBalancer address existed in Kubernetes, but it was not the address external clients used—and it was not necessarily an intermediate hop in that hostPort path.

The private Service VIP and the public endpoint describe different parts of the deployment. The public packet path does not automatically follow the Service's LoadBalancer address.

This is the ownership boundary worth removing. A controller watching the Service could discover 10.56.0.8, but it could not infer the public address from that field alone. The missing association lived outside Kubernetes.

Giving the Service its real public address eliminates that translation in the management model. It does not require making Pod addresses or ClusterIP addresses public.

What Actually Changes Between the Two Stacks

The comparison is between two deployments, not between caricatures of Calico and Cilium.

The previous stack used Calico for Pod networking and BGP, MetalLB for LoadBalancer allocation, and kube-proxy for Service forwarding. In Calico's architecture, Felix programs the local dataplane while BGP is handled separately. Calico also supports an eBPF dataplane^[3], Service advertisements with local-endpoint semantics^[4], and LoadBalancer IPAM^[5]. Public Service addresses and removing MetalLB are therefore not capabilities exclusive to Cilium.

Cilium's advantage in this deployment is the replacement of the existing combination with a coherent set of controllers and a single Service datapath implementation:

These are control relationships, not a packet-processing pipeline. Packets do not pass through the IPAM controller or the BGP speaker.

MetalLB becomes unnecessary because its allocation responsibility has a replacement. Kube-proxy becomes unnecessary because its forwarding responsibility has a replacement. Neither disappears merely because BGP has been enabled. Cilium LB-IPAM^[6] and kube-proxy replacement^[7] address those different functions.

Cilium's BGP Control Plane is also deliberately narrower than a general-purpose router. It advertises reachability; it does not install the cluster's forwarding datapath or replace the underlay routing system. The node still needs working interfaces, routes, and a usable path to its peers. Cilium BGP architecture^[8].

A Reconstructed Network

The reference environment has a three-node control plane, a dedicated etcd tier, and twelve workers. A subset of workers participates in external Service routing. Two edge routers provide the boundary between the private underlay and the upstream network.

For node-facing BGP sessions, the routers use private ASN 65000 and the selected workers use private ASN 65100. These are internal peering identifiers, not the public origin ASN of the illustrative allocation.

Dashed links show BGP control relationships; solid arrows show representative inbound paths. The private underlay and management links are simplified.

An overlay is compatible with this design. Public Service routing does not require the physical network to learn every Pod prefix. Here, the edge learns Service host routes, while the Pod network remains private.

Restricting BGP to selected nodes is a design choice. With externalTrafficPolicy: Local, those nodes must also host eligible endpoints for the Services they expose. For a shared ingress Service, that means ingress-controller Pods—not every application behind the ingress controller. Cilium's node selection and Service advertisement resources^[9] express these relationships.

Full eBPF Means a Different Service Datapath

The final state has kube-proxy removed, not running alongside Cilium as the permanent Service implementation.

In RKE2, the relevant configuration spans both the distribution and the Cilium chart. The following excerpts express the reference state; they are not an in-place migration procedure:

# RKE2 configuration excerpt
cni: cilium
disable-kube-proxy: true

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    kubeProxyReplacement: true
    k8sServiceHost: "10.12.0.10"
    k8sServicePort: "6443"
    routingMode: tunnel
    tunnelProtocol: vxlan
    ipam:
      mode: kubernetes
    bgpControlPlane:
      enabled: true

The API endpoint must be reachable before Service forwarding is available; pointing this bootstrap dependency at an address that itself requires the missing Service datapath would be circular. RKE2 documents the distribution-level switch and the matching Cilium chart configuration^[10]. Chart values must match the version bundled with the RKE2 installation.

For this article, full eBPF refers to Cilium owning Service forwarding. It is not a claim that every packet on the host bypasses the Linux network stack, that XDP acceleration is enabled, or that all NAT has disappeared. Cilium's forwarding modes^[7:1] and masquerading configuration^[11] remain explicit choices.

Make the Public Address a Service Property

The public pool is a Kubernetes resource with an explicit selector:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: public-services
spec:
  blocks:
    - cidr: 20.72.64.0/20
  serviceSelector:
    matchLabels:
      networking.example.com/exposure: public

An application can then request public exposure without selecting an address manually. This example assumes an existing applications namespace and TLS-serving Pods selected by app: api:

apiVersion: v1
kind: Service
metadata:
  name: public-api
  namespace: applications
  labels:
    networking.example.com/exposure: public
  annotations:
    external-dns.kubernetes.io/hostname: api.apps.example.com
    external-dns.kubernetes.io/ttl: "60"
spec:
  type: LoadBalancer
  loadBalancerClass: io.cilium/bgp-control-plane
  allocateLoadBalancerNodePorts: false
  externalTrafficPolicy: Local
  selector:
    app: api
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443

The examples use the documented cilium.io/v2 resources. LB-IPAM^[6:1] assigns the LoadBalancer address; the class identifies the intended Cilium exposure mechanism. The API server still allocates a separate ClusterIP.

An illustrative result is:

NAME         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)
public-api   LoadBalancer   10.96.0.18     20.72.64.17    443/TCP

20.72.64.17 is an example allocation, not a promise about allocation order. The address is written by the controller to status.loadBalancer.ingress; it is not a status block that the application author should maintain in Git. A required reservation can instead be requested through lbipam.cilium.io/ips. Address requests^[6:2].

One assigned public identity, materialized in three different systems.

The internal address has not vanished. 10.96.0.18 remains the ClusterIP, and the backends retain private Pod addresses. What disappears is the separate mapping between a private LoadBalancer VIP and an independently maintained public VIP.

BGP Connects the Service to the Physical Network

The reference BGP policy selects public Services and selected speaker nodes. It does not export the Pod CIDR or the ClusterIP range.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: public-service-routes
  labels:
    routing-scope: public-services
spec:
  advertisements:
    - advertisementType: Service
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchLabels:
          networking.example.com/exposure: public
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: edge-peers
spec:
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          routing-scope: public-services
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: public-service-speakers
spec:
  nodeSelector:
    matchLabels:
      networking.example.com/bgp-speaker: "true"
  bgpInstances:
    - name: service-routing
      localASN: 65100
      peers:
        - name: edge-a
          peerASN: 65000
          peerAddress: 10.24.0.1
          peerConfigRef:
            name: edge-peers
        - name: edge-b
          peerASN: 65000
          peerAddress: 10.24.0.2
          peerConfigRef:
            name: edge-peers

These resources describe different selections: which nodes establish sessions, which advertisement objects a peer uses, and which Services those advertisements match. The corresponding router configuration must accept the sessions and install the permitted paths. BGP resource model^[9:1].

Host routes inside, aggregate routing outside

There are two routing scopes. The upstream network delivers traffic for the provisioned public aggregate to the edge. Inside the data center, Cilium advertises exact Service routes to the edge. A Service /32 is not being exported as a standalone route throughout the Internet.

In this design, each edge accepts only Service /32s within 20.72.64.0/20 from the cluster, enables an appropriate ECMP width, and has a discard route covering that Service pool. The more-specific active host routes win; unallocated or withdrawn addresses do not fall through to an upstream default route and loop back. This is an application of longest-prefix forwarding^[12], not a requirement for Cilium to announce an aggregate itself.

The public pool is deliberately not an on-link subnet on the worker VLAN. For a route such as:

20.72.64.17/32 via 10.24.0.11

the edge resolves the next-hop node address, not the Service VIP, to a link-layer address. The IP destination remains 20.72.64.17 until Service processing on the node. IPv4 next-hop forwarding^[12:1].

This also narrows the ARP discussion in the earlier article: on-link VIP ownership and routed next-hop reachability are different cases. An unbound VIP is not, by itself, evidence that a routed BGP Service design is broken. This reference topology does not require proxy ARP for every Service address.

Local endpoints determine the ingress paths

For a Service using externalTrafficPolicy: Local, Cilium stops advertising from a node when that node has no local endpoint. Consequently, the upstream ECMP set can follow endpoint placement rather than include every worker indiscriminately. Service advertisement policy^[9:2].

The BGP session and an individual Service advertisement have separate lifecycles.

This is not instantaneous failure detection or connection-state replication. Route withdrawal takes time to reach the forwarding table, and ECMP changes can move existing flows to another node. Cilium's operation guide^[13] explicitly describes the resulting connection-reset cases.

eBPF Completes the Packet Path

BGP brings the packet to an eligible node. Cilium's Service datapath then selects a backend. The router's ECMP decision and the node's backend-selection decision are two separate operations.

The figure illustrates a local-backend flow. It does not imply DSR or a remote backend bypassing the ingress node.

The edge no longer needs a per-Service DNAT or server-load-balancing object. That does not make the entire path NAT-free: translating a Service frontend to a Pod backend is still part of Service processing. Remote-backend return paths depend on whether SNAT, DSR, or another supported forwarding mode is selected. Cilium Service forwarding^[7:2].

The useful simplification is therefore precise: there is no separate public-to-private load-balancer mapping at the edge, and there is no kube-proxy Service datapath on the node.

ExternalDNS Can Consume the Address Without a Translation Layer

Once the LoadBalancer status contains the real public address, DNS automation becomes straightforward. The desired hostname comes from the Service; the target comes from its assigned LoadBalancer status.

For this Service shape—without a target override or spec.externalIPs—ExternalDNS's Service source^[14] derives its target from status.loadBalancer.ingress. It does not need to inspect the edge appliance or maintain a second public-address mapping.

A reference ExternalDNS argument set is:

args:
  - --source=service
  - --service-type-filter=LoadBalancer
  - --label-filter=networking.example.com/exposure=public
  - --annotation-prefix=external-dns.kubernetes.io/
  - --provider=google
  - --google-project=example-public-dns
  - --google-zone-visibility=public
  - --domain-filter=apps.example.com
  - --registry=txt
  - --txt-owner-id=cluster-reference-a
  - --txt-prefix=_externaldns-
  - --policy=sync
  - --interval=1m

This is a controller-argument excerpt, not a complete Deployment. The provider and filtering flags^[15] are explicit, including the annotation prefix used in the Service example. Older deployments using the external-dns.alpha.kubernetes.io/ prefix must keep their annotations and controller configuration aligned.

The integration assumes an existing, correctly delegated public Cloud DNS zone and an authenticated identity allowed to manage its records. Running RKE2 outside Google Cloud does not require turning it into GKE; the Google provider supports an explicit project, and Workload Identity Federation for Kubernetes^[16] provides an authentication option for external clusters.

The TXT registry gives this controller an ownership marker. A unique owner ID and a restricted domain scope matter when sync is allowed to delete records that disappear from the desired state. They prevent accidental overlap between controllers, but do not replace IAM authorization. ExternalDNS TXT ownership^[17].

The resulting record is simply:

api.apps.example.com.  60  IN  A  20.72.64.17

There is no engineer-maintained step that translates a private Service VIP into this value. The relationship is recoverable from Kubernetes state.

Shared ingress still fits

Public Service addresses do not imply one public IP per application. A Traefik LoadBalancer Service can own one public address while many HTTP applications share it.

For this pattern, ExternalDNS can use its Ingress source^[18]. Traefik must publish the fronting Service's LoadBalancer address into Ingress status; its published-Service configuration^[19] provides that bridge. ExternalDNS should not be given a hardcoded replacement target that recreates the original synchronization problem.

The public IP belongs to the ingress Service. Application hostnames belong to their routing resources. Both identities remain visible inside the cluster.

One Source of Intent, Several Independent Controllers

“Kubernetes is the network control plane” does not mean that Kubernetes replaces every network system. It means that the application-facing declaration originates there, within a previously provisioned network boundary.

The platform still owns address entitlement, upstream routing, prefix filters, firewall policy, DNS-zone delegation, and controller credentials. A label that requests public exposure must also be governed by RBAC and admission policy; it is not an authorization boundary on its own.

Inside that boundary, the Service is enough to drive address allocation, routing, forwarding, and DNS publication. The router does not need an application-specific object, and the DNS record does not need an independently entered target.

These systems converge rather than commit atomically. An assigned address can be visible before a router installs its route. A DNS update can complete before the datapath is ready. Endpoint loss can withdraw the last Local route while the Service retains its address and DNS still resolves. Service-source behavior^[14:1] and BGP endpoint-loss behavior^[13:1] make those distinct states important.

Deleting a Service has the same property: route withdrawal, address release, and DNS reconciliation are not one distributed transaction. Resolver caching also outlives an authoritative DNS change. Address reuse and retirement therefore remain lifecycle concerns even when no operator has to edit the individual records.

Three Failures at the Migration Boundary

The final topology is cleaner than the path used to reach it. Three incidents exposed state that was not represented by the new Service and Cilium manifests.

These cases occurred during the transition. In particular, the HostPort incident below belongs to the temporary portmap-based path, before the final kube-proxy-free eBPF Service state.

1. RKE2's Active Configuration Was Still in Memory

After changing the CNI setting on disk, manually placing the Cilium manifest in RKE2's manifests directory did not start the intended deployment. RKE2 deleted the file again.

The running server process still held the previous CNI selection. The file on disk had changed; the process enforcing the desired add-on set had not reloaded it.

Restarting the server made the configuration effective and allowed RKE2 to generate the selected CNI manifest itself. The failure was not a malformed Cilium manifest. It was a disagreement between configuration on disk and the active configuration of the controller managing that manifest.

2. The Network Bootstrap Was Blocked by Scheduling

Removing Calico left nodes with the node.kubernetes.io/network-unavailable:NoSchedule taint. The operator configuration used during this migration did not tolerate it, leaving the Cilium operator Pending while network initialization was already stalled.

Clearing the stale taint during the controlled recovery unblocked the operator and agent initialization. This was an observed interaction between the transition state and the deployed tolerations—not a claim that every Cilium installation has this dependency, or a recommendation to remove network taints indiscriminately.

The important distinction is between steady-state scheduling policy and bootstrap dependencies. The scheduling rule intended to keep workloads away from an unavailable network was also preventing a component needed for recovery from starting.

3. The Previous Network Survived in HostPort DNAT Rules

The most misleading failure appeared after replacement Pods were already healthy. A direct request to the ingress Pod succeeded, but requests to worker hostPorts did not.

The node still had an older portmap DNAT rule ahead of the replacement rule:

CNI-HOSTPORT-DNAT
  1. old network: k8s-pod-network
     TCP :80 -> 10.64.210.42:8000   [removed Pod]

  2. new network: portmap
     TCP :80 -> 10.64.21.18:8000    [running Pod]

The migration notes traced the incomplete cleanup to the changed CNI network identity: the old rules belonged to k8s-pod-network, while the replacement configuration used portmap. Whatever Kubernetes reported about the new Pods, the installed legacy translation still determined where hostPort traffic went.

Removing the obsolete entries, while preserving the replacement rules, restored the transition path. This was not a need to retain portmap in the final architecture. It was a need to remove state left behind by the previous owner.

A related check identified Pods that still carried addresses from the old IPAM state. With Kubernetes host-scope IPAM^[20], an ordinary IPv4 Pod should receive an address from its node's allocated Pod CIDR. A Pod outside that range is a definite mismatch in this configuration:

Node PodCIDR:  10.64.21.0/24
Old Pod IP:   10.64.210.42  -> outside the node allocation
New Pod IP:   10.64.21.18   -> inside the node allocation

This is a negative check, not complete proof of migration. Host-network Pods must be excluded, and an address inside the expected range does not by itself prove that Cilium owns the endpoint. Running is a container-lifecycle observation, not a complete statement about network ownership.

The Public Endpoint Is Now a Kubernetes Object

The architectural result is not simply fewer components. It is fewer independently maintained descriptions of the same public service.

The address in Kubernetes can be the address clients actually use. The physical network learns which nodes can receive traffic for it. Cilium implements the Service datapath without kube-proxy. ExternalDNS can derive the corresponding Google Cloud DNS record from the same Kubernetes state.

Calico, MetalLB, and kube-proxy were implementation choices in the previous deployment. The more consequential boundary was between the cluster's private description of a service and the infrastructure's public description of it.

Once address ownership, upstream routing, and platform policy are established, publishing an application becomes a Kubernetes resource change—not a second, manually synchronized network deployment.