Proxmox VE Clusters

Choose Proxmox VE when the cluster has to run on hardware you already own, on-premises or on dedicated servers. You bring the Proxmox nodes, storage and bridges; Ankra fills in safe defaults where they are missing, creates the VMs, installs Kubernetes, and then operates the cluster for you - this is an Ankra Managed cluster. There is no cloud bill: Cloud Cost prices the cluster from a rate card on its Proxmox credential, which starts with Ankra’s default rates.

Before you start

A Proxmox VE API credential: the API URL and an API token with VM management privileges. If Ankra needs to initialise storage or networking, the token also needs Datastore.Allocate, Sys.Modify and SDN.Use. See Proxmox VE Credentials.
An SSH key credential for VM access - your own public key, or one Ankra generates. A Proxmox cluster takes a single key. See SSH Key Credentials.
A cloud-init VM template. If a node has none, Ankra creates one during credential setup: it downloads the Ubuntu 24.04 cloud image and registers it as the ankra-ubuntu-24-04 template. You can bring your own and pin it with template - see Creating a cloud-init template.
A network bridge. Use an existing active Linux bridge, or let Ankra convert the single active management Ethernet or bond interface (static or manual method) to vmbr0. Ankra also creates a private NAT network ankra (10.20.0.0/16) through Proxmox SDN - see Networking.
The dnsmasq package on each node for the private network’s DHCP (apt install dnsmasq && systemctl disable --now dnsmasq), and source /etc/network/interfaces.d/* in /etc/network/interfaces (present by default on standard installs).
Free local disk or existing VM storage. If no active storage supports VM images, Ankra enables images and rootdir on the local directory storage, or creates it at /var/lib/vz. The import content type is enabled on local for cloud-image downloads.
API reachability. The Proxmox API must use HTTPS and be reachable from Ankra, directly or through an SSH jumphost.

Hybrid connectivity (SSH jumphost)

Many Proxmox environments are not reachable from the internet. If Ankra cannot reach the Proxmox API and the VM network directly, attach an SSH jumphost (host, port, username and private key) to the Proxmox VE credential; Ankra then tunnels both the API calls and the SSH connections to your nodes through it. If the API uses a self-signed certificate, enable TLS insecure (tls_insecure) on the credential. Both are set on the credential - see Proxmox VE Credentials.

Create the cluster

Dashboard
CLI
API

Open the create dialog

Go to Clusters, click Create cluster and pick Proxmox VE under Ankra Managed.

Credential

Pick the Proxmox VE credential and an SSH key, or add either inline.

Placement

Pick Single host and the Proxmox node that hosts all cluster VMs, or Host spread and two or more hosts - see Host-Spread Placement. In both modes pick the Storage for VM disks and the Network Bridge the VMs attach to. Pick the ankra private network unless you run your own DHCP-served bridge.

Compute

Pick the bastion size (for example px-small), the control plane count (1 or 3) and size (for example px-medium), worker node groups, and for kubeadm the etcd topology. The wizard shows each size’s vCPUs, memory and disk.

Kubernetes

Keep kubeadm (the default, with Cilium) or pick k3s and its CNI, and optionally the version. See Choices fixed at create time.

Review

Name the cluster and set its environment. Two checkboxes are on by default:

Include Networking Stack - Traefik and cert-manager as Ankra-managed stacks. On k3s they run through the built-in service load balancer; unchecked, k3s keeps its bundled Traefik and no certificate manager (a kubeadm cluster gets no ingress controller or certificate manager at all).
Include Public DNS - a delegated subdomain on ankra.cc with external-dns wired, so an ingress hostname under it gets its DNS record and TLS certificate automatically.

Click Create cluster. A progress view follows the SSH keys, bastion, VMs, Kubernetes installation and Ankra Agent.

ankra credentials proxmox create --name my-proxmox --api-url https://pve.example:8006 --token-id 'root@pam!ankra'   # prompts for the token secret
ankra credentials proxmox ssh-key create --name my-ssh-key --generate

# What the credential can use
ankra cluster proxmox hosts --credential-id <proxmox-credential-id>
ankra cluster proxmox storages --credential-id <proxmox-credential-id> --node pve01
ankra cluster proxmox bridges --credential-id <proxmox-credential-id> --node pve01
ankra cluster proxmox templates --credential-id <proxmox-credential-id> --node pve01
ankra cluster proxmox sizes

ankra cluster proxmox create \
  --name my-cluster \
  --credential-id <proxmox-credential-id> \
  --ssh-key-credential-id <ssh-key-credential-id> \
  --node pve01 \
  --storage local-lvm \
  --bridge ankra \
  --control-plane-count 1 \
  --control-plane-instance-type px-medium \
  --worker-count 2 \
  --worker-instance-type px-medium

--placement-nodes pve01,pve02,pve03 spreads the VMs across hosts. --template, --distribution k3s, --etcd-topology external with --etcd-node-count and --etcd-instance-type, and --include-networking=false cover the other choices. The credential flags --tls-insecure and --jumphost-* set the options from Hybrid connectivity.

curl -X POST https://platform.ankra.app/api/v1/clusters/proxmox \
  -H "Authorization: Bearer $ANKRA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-cluster",
    "credential_id": "<proxmox-credential-id>",
    "ssh_key_credential_id": "<ssh-key-credential-id>",
    "node": "pve01",
    "storage": "local-lvm",
    "bridge": "ankra",
    "control_plane_count": 1,
    "control_plane_instance_type": "px-medium",
    "node_groups": [
      {"name": "default", "instance_type": "px-medium", "count": 2}
    ],
    "include_networking": true,
    "include_dns": true
  }'

Every field, default and instance size preset is in the Proxmox VE Reference.

Verify

The cluster moves from Provisioning to Online in the clusters list once the Ankra Agent has connected.
Point kubectl at it through Ankra (this needs a Cluster Access grant) and check the nodes:
ankra cluster kubeconfig add my-cluster --use kubectl get nodes
Every control plane and worker should be Ready. See Accessing Clusters with kubectl.

What Ankra created on your hosts

A bastion VM - the only SSH access point Ankra uses to reach the nodes. It carries no workload or egress traffic. With a jumphost on the credential, Ankra reaches both the Proxmox API and the bastion through it.
Control plane and worker VMs (QEMU), cloned from the cloud-init template and attached to the network you picked, plus dedicated etcd VMs for a kubeadm cluster with external etcd.
The vmbr0 bridge and the ankra SDN network, when they did not exist, set up during credential setup.

There is no cloud controller manager: LoadBalancer Services are not provisioned. Expose workloads with NodePort Services, an ingress controller, or a load balancer you deploy yourself. The Proxmox VE Reference has the full layout.

Proxmox VE specifics

Networking

When you save a Proxmox VE credential, Ankra initialises two networks when they do not already exist:

Network	Purpose	Address
`vmbr0` (Linux bridge)	Management - carries the host’s public or provider-assigned IP	Inherited from the physical interface
`ankra` (SDN vnet)	Private compute - isolated from the management interface	10.20.0.1/16 (SNAT to the public interface)

vmbr0 (management bridge). If no active bridge exists, Ankra converts the single active Ethernet or bond management interface to vmbr0. A statically addressed interface’s address and gateway move to the bridge; a manual-method interface (common on dedicated-server providers such as Scaleway) converts to an address-less bridge. The physical interface becomes the bridge’s uplink. DHCP-managed management interfaces cannot be converted through the Proxmox API - create the bridge manually in that case. Ankra refuses the conversion when the management interface is ambiguous, has custom options, or another network change is pending. ankra (private SDN network). Ankra creates a Proxmox SDN simple zone named ankra with a vnet of the same name and the subnet 10.20.0.0/16 (gateway 10.20.0.1). The subnet has SNAT enabled, so Proxmox masquerades outbound VM traffic through the host’s public interface, and DHCP served by dnsmasq (range 10.20.1.1 to 10.20.254.254) backed by the built-in PVE IPAM. Because addresses are registered in the IPAM the moment a VM’s NIC attaches, Ankra reads VM addresses without requiring the QEMU guest agent in the image. Recommended setup. Place cluster VMs on the ankra network - it appears alongside Linux bridges in the cluster creation wizard. This keeps Kubernetes traffic off the management bridge and gives VMs internet access through NAT without exposing them publicly. Because the VMs are private, attach the Proxmox host itself (or another host with a route into 10.20.0.0/16) as the SSH jumphost on the credential so Ankra can reach the bastion.

Creating a cloud-init template

Ankra clones a cloud-init VM template for every cluster VM. Nodes without any template get one automatically: during credential setup Ankra downloads the Ubuntu 24.04 cloud image and registers it as the ankra-ubuntu-24-04 template. Nodes that already carry a template are left untouched, so a custom template always wins. Create your own template when you need a different distribution, a hardened image, or pre-installed packages. The steps below use Ubuntu 24.04. Adjust the image URL and VM ID for your environment.

# On the Proxmox node (as root)

# 1. Download the Ubuntu 24.04 cloud image
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img \
  -O /var/lib/vz/images/ubuntu-2404-cloud.img

# 2. Create a VM to use as the template (VM ID 9000, adjust as needed)
qm create 9000 \
  --name ubuntu-2404-cloudinit \
  --memory 2048 --cores 2 \
  --net0 virtio,bridge=ankra \
  --ostype l26

# 3. Import the cloud image as the primary disk
qm importdisk 9000 /var/lib/vz/images/ubuntu-2404-cloud.img local-lvm

# 4. Attach the disk, enable SCSI, and configure boot
qm set 9000 \
  --scsihw virtio-scsi-pci \
  --scsi0 local-lvm:vm-9000-disk-0,discard=on \
  --boot c --bootdisk scsi0

# 5. Add a cloud-init drive and serial console
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --serial0 socket --vga serial0

# 6. Enable the QEMU guest agent (required for Ankra to read VM IPs on
#    networks other than the Ankra-managed SDN network)
qm set 9000 --agent enabled=1

# 7. Convert the VM to a template
qm template 9000

The template VM ID (9000 in this example) must be unique on your node. Ankra auto-picks the template with the lowest VMID unless you pin one with the template option at cluster creation time.

For host-spread placement, create the same-named template on every selected node before provisioning the cluster.

Host-Spread Placement

When your Proxmox VE cluster has multiple nodes, you can spread the Kubernetes VMs across them so a single host failure does not take down the whole Kubernetes cluster. Host spread is opt-in: pass placement_nodes with two or more nodes via the API, or pick Host spread in the creation wizard.

curl -X POST https://platform.ankra.app/api/v1/clusters/proxmox \
  -H "Authorization: Bearer $ANKRA_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-cluster",
    "credential_id": "<proxmox-credential-id>",
    "ssh_key_credential_id": "<ssh-key-credential-id>",
    "node": "pve01",
    "placement_nodes": ["pve01", "pve02", "pve03"],
    "storage": "local-lvm",
    "bridge": "ankra",
    "control_plane_count": 3,
    "control_plane_instance_type": "px-medium",
    "node_groups": [
      {"name": "default", "instance_type": "px-medium", "count": 3}
    ],
    "distribution": "k3s"
  }'

Prerequisites

Every selected host must be online and provide the same-named resources, because VMs are cloned locally on each host:

Active storage with the selected name that allows VM images (e.g., local-lvm on every host).
An active bridge with the selected name (e.g., vmbr0 on every host), all attached to the same DHCP network.
A cloud-init template with the same name on every host. Ankra resolves each host’s own template VMID and clones locally - shared-storage cross-node cloning is not used.

Creation is rejected if any selected host is missing one of these. node stays required and must be one of the placement_nodes; the bastion always runs there.

How VMs are distributed

Ankra assigns control planes, dedicated etcd members, and each worker group deterministically: the host with the fewest members of the same role or group wins, ties break by fewest total Kubernetes VMs, then by your selected-node order. Every Kubernetes node is labeled with topology.kubernetes.io/zone=<proxmox-node> so you can use standard topology spread constraints and pod anti-affinity against the physical failure domain. Later scaling keeps the policy: adding node groups, scaling them up, and adding control planes all place new VMs on the least-represented eligible host. Restart, resize, upgrade, and delete operations stay pinned to each VM’s recorded host.

Requirements and limitations

At least 3 control planes. Host-spread clusters must be created with 3 or 5 control plane nodes so the Kubernetes control plane keeps quorum when one host fails.
Two hosts give reduced resilience. With only two hosts, losing the host that carries the control-plane majority still takes the Kubernetes API down. Three or more hosts are recommended.
The bastion is not spread. It runs on the primary node; if that host fails, Ankra’s SSH path to the cluster is unavailable until the host returns, while workloads keep running.
No Proxmox HA-manager integration. Ankra does not enroll VMs in Proxmox HA groups and never live-migrates or recreates VMs on another host after a failure. Resilience comes from Kubernetes replication across hosts, so run workloads with multiple replicas spread over zones.

Workload guidance

For a workload to survive a host outage, run at least two replicas and spread them across zones:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

Other differences

Resizing a node group powers each VM off, resizes it and powers it on again, with brief downtime for its workloads. Sizes only grow; for smaller nodes, create a new group and delete the old one. On a host-spread cluster, new VMs go to the least-represented eligible host.
Restart works from the CLI (ankra cluster proxmox nodes restart); the bastion is resized from the dashboard or the API.
A stop captures the cluster’s etcd state through the credential’s jump host before the VMs are deleted, and frees the host capacity. Terminating deletes every VM Ankra created (bastion, control planes, workers and etcd VMs) and leaves your Proxmox nodes, storage and bridges untouched.
No cost estimate in the create wizard, because Proxmox VE has no list pricing. Cloud Cost prices running clusters from the credential’s rate card, which ships with default rates you can change.

Operate it

Day-2 tasks work the same way on every Ankra Managed provider, with the CLI name proxmox:

Troubleshooting

Issue	Solution
Proxmox API unreachable from Ankra	Attach an SSH jumphost to the credential - Ankra tunnels API calls and node SSH through it
TLS certificate errors (self-signed)	Enable the TLS insecure toggle (`tls_insecure`) on the Proxmox VE credential
Nodes never get an IP address	On the `ankra` SDN network, verify the zone is healthy (`dnsmasq` installed, zone status available under Datacenter → SDN). On your own bridge, verify a DHCP server answers there
Provisioning fails waiting for a VM’s IP address	On the `ankra` SDN network, addresses come from the PVE IPAM automatically. On other bridges the cloned template must ship the QEMU guest agent - rebuild the template with `qemu-guest-agent` and retry
No VM templates found on the node	Re-save the Proxmox credential to run the automatic setup (it creates the `ankra-ubuntu-24-04` template), or create one yourself - see Creating a cloud-init template
Cluster stuck in provisioning	Check the API token privileges (see Proxmox VE Credentials) and free capacity on the selected node and storage
VM creation fails with a permission error	The API token is missing privileges (e.g., `VM.Allocate`, `Datastore.AllocateSpace`) on the selected node, storage, or bridge
LoadBalancer service stuck in `Pending`	Proxmox clusters have no cloud controller manager - use `NodePort`, ingress, or your own load balancer
No cost estimate in the wizard	Expected - Proxmox has no list pricing. Running clusters are priced in Cloud Cost from the credential’s rate card

Get Started

Concepts

Create and Run Clusters

Ship Your App

Stacks and Add-ons

Observe and Troubleshoot

Ankra AI

Connect Tools

Source Control

Security and Access

Cost and Backups

Organisation and Account

Reference

Before you start

Hybrid connectivity (SSH jumphost)

Create the cluster

Verify

What Ankra created on your hosts

Proxmox VE specifics

Networking

Creating a cloud-init template

Host-Spread Placement

Prerequisites

How VMs are distributed

Requirements and limitations

Workload guidance

Other differences

Operate it

Troubleshooting

​Before you start

​Hybrid connectivity (SSH jumphost)

​Create the cluster

​Verify

​What Ankra created on your hosts

​Proxmox VE specifics

​Networking

​Creating a cloud-init template

​Host-Spread Placement

​Prerequisites

​How VMs are distributed

​Requirements and limitations

​Workload guidance

​Other differences

​Operate it

​Troubleshooting

Before you start

Hybrid connectivity (SSH jumphost)

Create the cluster

Verify

What Ankra created on your hosts

Proxmox VE specifics

Networking

Creating a cloud-init template

Host-Spread Placement

Prerequisites

How VMs are distributed

Requirements and limitations

Workload guidance

Other differences

Operate it

Troubleshooting