Skip to main content
When a service runs in one cluster and its callers run in another - a second region, a second provider, an on-prem site - you normally have to expose it publicly or build your own VPN between the clusters. Cluster Mesh connects two or more of your clusters into one Cilium ClusterMesh instead: pods reach pods across clusters, and a global service resolves to healthy backends wherever they run, over an encrypted WireGuard overlay Ankra manages.

​
Overview

A mesh is an organisation-level object with clusters as members. When a cluster joins, Ankra handles the whole configuration: it creates the mesh’s shared certificate authority on the first join and hands it to every member, updates each member’s Cilium with its mesh identity and peer list, deploys the clustermesh-apiserver, and gives every node a WireGuard peer, and a route, for every node of every other member. Nothing about the mesh requires exposing your Kubernetes API or pod network to the internet: peers talk through the platform overlay. What you can do with a connected mesh:
  • Global services: annotate the same Service in two clusters with service.cilium.io/global: "true" and requests resolve to healthy backends in either cluster.
  • Cross-cluster pod traffic: pods reach pod IPs in other members directly.
  • Network policy across clusters: Cilium policies can select peers in other members by cluster name.
Watch two clusters in two countries joined into one network.

​
Requirements

Mesh requirements fall into three groups, and the group a requirement is in tells you what to do about it: one group is settled when the cluster is built, one is fixable on a running cluster, and one is about the clusters you mesh together rather than any one of them.

​
Settled when the cluster is built

These four are baked into a cluster and no operation changes them. A cluster that fails any of them can only mesh if you rebuild it. Readiness reports the provider and distribution under its platform check, the CNI under cni, and the managed Cilium release under cilium_stack, each marked not remediable.

​
Fixable on a running cluster

A cluster that clears the four above is mesh-capable, which is not the same as ready. Two more things must be true, and both can be arranged in place with make-ready:
  • A Cilium cluster identity. An id in the range 1 to 255, and a name unique across the mesh. Clusters created while the WireGuard overlay was enabled for your organisation already have one. Anything older gets one allocated by make-ready on the running cluster.
  • A routable node network. A mesh forwards pod traffic to remote node addresses, so nodes on a per-cluster private network nothing else reaches can never carry it. One of three networks qualifies: the platform WireGuard overlay (--network-mode wireguard_mesh), Proxmox’s EVPN network, or a tailnet the nodes advertise.
make-ready itself needs the cluster to be online. A cluster still provisioning would race its own create jobs, and one mid-update could interleave with the update in flight, so wait for it to settle. Proxmox clusters additionally need --site-public-ip (see Proxmox sites).
One case make-ready cannot serve: an older cluster whose network identity was allocated without an overlay range. make-ready refuses it with that reason rather than half-preparing it. Contact support if you need such a cluster in a mesh.

​
Requirements across the members

These are properties of a mesh, not of a cluster, so they can only fail once you try to put two particular clusters together.
  • No overlapping pod ranges. The pod CIDR of every member must be disjoint from every other member’s: a mesh routes pod traffic between members by pod address. Service ranges may overlap - a ClusterIP is resolved inside the cluster that owns it and never crosses to a peer - so the kubeadm default service range is fine. Clusters created while the overlay is enabled draw disjoint pod ranges from your organisation’s pools automatically, and the platform keeps pod ranges unique across the organisation. Two clusters built before the overlay both run the kubeadm default pod range, so make-ready on the second one is refused by name, and moving a running cluster onto a fresh range is disabled for now (see Moving a cluster onto a fresh pod range) - so that pair cannot mesh with each other yet.
  • Unique Cilium cluster ids. Every member holds a different one, which caps a mesh at 255 members.
  • A routable path per pair. Connecting two apiservers is not enough. Ankra resolves each pair to one address family and refuses pairs with no route (see How members connect). In short: two clusters in different provider accounts must both be on the WireGuard overlay, and a Proxmox EVPN cluster meshes only with clusters on the same Proxmox credential.
  • Each member publishes an apiserver address. A member publishes one once the platform has recorded its nodes, so this is not checked before a join (it would refuse a cluster that is minutes from ready). It is checked afterwards, and a member nobody can reach is reported rather than left silently unconnected.
  • One mesh per cluster. A cluster belongs to at most one mesh; joining a second is refused.

​
On the platform side

  • Permissions. Reading meshes and running readiness needs clusters.read; creating a mesh, joining, leaving and make-ready need clusters.write. Both are checked at organisation scope, because admitting a cluster changes the configuration every other member runs with.
  • A fixed NodePort. Every member publishes its clustermesh-apiserver on NodePort 32379, the same across the fleet so a peer needs only an address to dial one. Keep it free, and let it through any firewall between members.
  • MTU is pinned to 1420 on mesh members, matching the overlay. Cilium’s VXLAN rides inside the overlay’s WireGuard frame, and a member configured above that black-holes pod traffic between clusters while everything inside each cluster keeps working.
  • Cilium’s own encryption is off on mesh members, deliberately. Every transport the mesh uses already carries packets encrypted, so Cilium’s WireGuard layer would encrypt a second time and shrink the MTU again for no gain.
  • The shared trust root comes first. A member’s mesh configuration is held back until its agent confirms the mesh certificate authority is installed. Without that, Cilium would create a certificate authority of its own, and the member would present a certificate no peer trusts while reporting a healthy mesh.

​
Where the Cluster Mesh section appears

A cluster’s sidebar carries a Cluster Mesh entry when Cluster Mesh is enabled for the organisation and that cluster can actually use it: it is already a mesh member, or it is eligible, meaning it can join now or after a make-ready. A cluster that fails one of the settled-at-build requirements has no entry, because nothing you could do from that page would change the answer. The organisation-level Cluster Meshes page lists every mesh regardless and stays reachable from the main menu. Eligibility is exactly what readiness reports, so the API and the CLI answer the same question the sidebar does:

​
Connect two clusters

1

Check readiness

Each cluster is reported ready, or every failing check is spelled out, including which failures are fixable on a running cluster and which are settled when the cluster is built.
2

Create the mesh and join both clusters

3

Watch it converge

The mesh and its members move pending → configuring → ready as the agents prove each step: identity installed, shared trust root in place, every expected peer connected. A healthy pair usually converges within 5 to 15 minutes.

​
What the statuses mean

Statuses are derived from what each cluster’s agent actually reported, never from what the platform intended, so ready means the connection was proven from inside the cluster.

​
Make an existing cluster mesh-capable

A cluster created without --network-mode wireguard_mesh (or before the overlay existed) can be made mesh-capable in place, with nothing deleted or recreated:
make-ready allocates the cluster’s Cilium identity and an overlay range, keeps the pod and service ranges the cluster already runs, and sets its nodes converging onto the overlay: each node installs WireGuard and joins the overlay through its own update job, the node IP switches to the overlay address, and Cilium re-renders with the identity. A cluster’s status view carries an overlay_nodes item while nodes are still joining, naming how many are left.
Zero disruption is not possible: switching node IPs restarts each kubelet once. Plan it like a rolling node operation. Everything else stays: workloads, data, and the addresses your Services expose.

​
Moving a cluster onto a fresh pod range

Moving a running cluster onto a different pod range is disabled. make-ready refuses it: kube-controller-manager exits at start while any node still holds a range outside --cluster-cidr, so the move takes the cluster’s controllers down instead of renumbering it. Two clusters that both run the kubeadm default pod range cannot mesh with each other yet.
Two clusters built before the overlay both run the kubeadm default pod range, and pod ranges must be disjoint across a mesh, so the platform keeps them unique per organisation: make-ready on the second such cluster is refused by name, and there is no renumber to fall back on for now. The one range --pod-cidr still accepts is the kubeadm default 10.244.0.0/16, which puts a cluster an earlier renumber moved off it back onto the range its nodes run:
That records the range on the identity and stamps it onto every node, and each control plane updates the settings that carry the range in place (the kubeadm configuration, the controller-manager’s --cluster-cidr, kube-proxy’s clusterCIDR). No node is drained, because every node already runs the range. A node whose range lags the definition reads as network=pod_cidr_drift until its update job has converged it. make-ready refuses, with the reason, clusters it cannot serve: providers not yet wired to the overlay, non-kubeadm distributions, non-Cilium CNIs, Proxmox clusters without a site address, a default pod range another cluster of the organisation already recorded, and any pod range that would move a running cluster off the one it runs.

​
The whole path in one command

ankra cluster mesh up runs the path above for a set of clusters: it reads their readiness, runs make-ready on every cluster that only lacks its identity or the overlay, joins every cluster that is ready, creates the mesh when none of that name exists, and keeps passing until every member reports ready or --wait runs out. It changes nothing with --plan.
--renumber-pods asks up to move every colliding cluster after the first onto a fresh pod range. That move is disabled for now, so the request is refused: a set whose clusters share a pod range cannot be brought up as one mesh yet.

​
How members connect

Ankra resolves each pair of members to one address family, taking the first that fits: The overlay wins over the others on purpose: it is the one network the mesh itself makes routable between two members, so it works across accounts, providers and sites alike. Two consequences worth knowing before you plan a mesh:
  • Two clusters in different provider accounts mesh only if both are on the WireGuard overlay. Otherwise their apiservers might connect while pod traffic has nowhere to go, which is why Ankra refuses the pair outright instead.
  • A Proxmox cluster on the EVPN network has nothing that turns a public endpoint into a routable node address, so it meshes only with clusters on the same Proxmox credential.
The clustermesh-apiserver is reached inside the tunnel on the peer’s overlay address, so nothing about a mesh requires exposing your Kubernetes API or pod network to the internet.

​
Proxmox and on-prem sites

Proxmox nodes sit on a private SDN with no public addresses, so a Proxmox cluster publishes one site gateway: its bastion, reached through the site’s public address on a mapped UDP port range. A Proxmox site needs, once:
  1. A public address for the site, passed as --site-public-ip at create or make-ready.
  2. A UDP port-range forward on the site’s edge to the Proxmox host, and on the host to the cluster’s bastion: 51820-52074. The platform maintains the per-node port mapping on the bastion itself.
  3. Jump access for the platform: the credential’s jumphost must allow TCP forwarding to the bastion’s dial address (the permitopen list of the platform’s key).
One reachable side is enough to bring tunnels up, because the unreachable site initiates outbound and the tunnel is held open with keepalives, but forwarding both sites is more robust.
Sites behind NAT in both directions still mesh, as long as at least one site’s UDP range is reachable from the other.

​
Try it: a global service

Deploy the same Service in both clusters, backends wherever you like:
A pod in either cluster that resolves echo.demo is served by healthy backends from both. Remove the deployment on one side and traffic shifts to the other. To prove cross-cluster traffic step by step before you rely on it, follow Verify a Cluster Mesh.

​
Leaving and deleting

A mesh with members cannot be deleted out from under them. Remove the members first so their configuration is torn down deliberately.

​
Network ranges and their limits

Every cluster Ankra builds for your organisation - on Hetzner, OVHcloud, UpCloud, DigitalOcean, Scaleway, AWS EC2, Proxmox, HPE Morpheus or Ankra Cloud - gets its own network identity from your organisation’s address pools when it is created: a Cilium cluster id, a pod range, a service range and an overlay range. Because no two of your clusters share a range, any two of them can join a mesh later without being renumbered. Whether or not you ever build a mesh, the identity sets two limits: Imported clusters and provider-managed clusters (DOKS, GKE, AKS, EKS and the others) take no identity and do not count. Deleting a cluster returns its ranges to the pool. A cluster created before your organisation had a network identity keeps its distribution’s default /16 pod range and holds 256 nodes. What happens at a limit:
  • A create past the cluster limit is refused with 409 and “No cluster network range is free in this organisation”, naming the pool. Deprovisioning a cluster you no longer need frees a range.
  • A create that plans more nodes than its range holds is refused with 409 before anything is provisioned. The count is the control planes plus every node group’s count, or its autoscaling maximum where autoscaling is on. The message names the range, the limit and the count you asked for.
  • A scale-up past the node limit is refused the same way: adding a node group, scaling one up, raising an autoscaling maximum, and a scale-up the cluster autoscaler asks for. No server is created, and the autoscaler backs off. Without this, a node past the limit would join the cluster and never become Ready.
Raising the limits. A running cluster’s pod range cannot be widened in place (see Moving a cluster onto a fresh pod range). If you need a cluster with more than 64 nodes, or more than 32 Ankra-built clusters at once, contact support before you create them. Ankra can change how your organisation’s pool is divided for clusters created from then on: larger ranges for fewer clusters (128 nodes each, 16 clusters) or smaller ranges for more clusters (32 nodes each, 64 clusters). Clusters you already run keep their ranges.

​
Troubleshooting

  • A member sits in configuring: the agent has not yet confirmed the trust root or a verify has not run; check the cluster is online and its operations list for the mesh resource. Ankra retries every five minutes, and sends the trust root again to a member whose agent has not confirmed it within fifteen minutes. A cluster created with networking off can never leave configuring - readiness refuses it under cilium_stack before the join, and the fix is a cluster created with networking on.
  • A member goes degraded: it verified fewer peers than expected. Check the peer cluster’s health first, then the site path (UDP forward, site address) if the pair crosses sites.
  • A readiness check is marked not remediable: for the provider, the distribution and the CNI that is final, and a new cluster is the only fix. For the identity and the node network, make-ready is the fix and readiness marks them remediable. A pod range that overlaps a member’s is not remediable either while moving a running cluster onto a fresh range is disabled.
  • make-ready says the pod range is already recorded by another cluster: another cluster of the organisation was prepared from the same kubeadm default first. Moving this one onto a fresh range is disabled for now, so the two cannot mesh with each other yet.
  • A member’s status view shows overlay_nodes not ready: some nodes have not joined the WireGuard overlay yet. Their update jobs put them on it - retry a failed one from the cluster’s operations, or run make-ready again to set them converging.
  • A node reads network=pod_cidr_drift: the range recorded for the cluster and the range the node runs differ. While the move is disabled, the node’s update job converges a control plane’s pod-range settings onto the range the nodes already run and refuses to move a node off the range it holds, naming the node and its range in the job’s output. The output also says when a control plane has not converged the range yet.
  • LoadBalancer Services on a meshed UpCloud cluster: node addressing changes on the overlay; see the UpCloud guide for the current behaviour of provider load balancers on meshed clusters.

​
Next steps