> ## Documentation Index > Fetch the complete documentation index at: https://docs.ankra.ai/llms.txt > Use this file to discover all available pages before exploring further. # AWS Clusters > Create, manage, and scale self-managed k3s or kubeadm clusters on EC2 instances, in a VPC Ankra creates or one you already own, with Ankra. export const CliVersion = ({since, command, note}) => { const latestStableCli = "0.16.0"; const parse = version => String(version).split(".").map(part => parseInt(part, 10) || 0); const requested = parse(since); const stable = parse(latestStableCli); let isPrerelease = false; for (let index = 0; index < 3; index += 1) { if (requested[index] > stable[index]) { isPrerelease = true; break; } if (requested[index] < stable[index]) { break; } } const containerStyle = { display: "flex", alignItems: "baseline", gap: "0.6rem", margin: "1rem 0", padding: "0.6rem 0.9rem", border: "1px solid rgba(128, 128, 128, 0.35)", borderRadius: "0.5rem", fontSize: "0.9em", lineHeight: 1.5 }; const pillStyle = { flex: "none", padding: "0.1rem 0.5rem", borderRadius: "999px", background: "rgba(128, 128, 128, 0.18)", fontFamily: "ui-monospace, SFMono-Regular, Menlo, monospace", fontSize: "0.85em", fontWeight: 600, whiteSpace: "nowrap" }; const keepTogether = { whiteSpace: "nowrap" }; return
CLI v{since}+ {command ? ankra {command} {" "} needs : The commands on this page need}{" "} the ankra CLI v{since} or later {isPrerelease ? {" "} - a pre-release today, so enable the{" "} beta channel before upgrading : null} . Check yours with{" "} ankra --version ; upgrade with{" "} ankra upgrade .{note ? {note} : null}
; }; Ankra provisions self-managed Kubernetes clusters on [Amazon EC2](https://aws.amazon.com/ec2/): a VPC with public and private subnets across one or three availability zones, NAT egress, a bastion in a public subnet, control plane and worker instances in the private subnets, k3s or kubeadm installed over SSH, and the AWS cloud controller manager and EBS CSI driver wired to per-cluster instance profiles. Ankra creates the network by default, as it does on every other self-managed provider; an account with an established network can adopt an existing VPC instead, and Ankra then owns only what it creates inside it. **Closed beta.** Self-managed AWS clusters are in closed beta. The workflow is stable but the surface may still change, and it is enabled per organisation on request - until it is, AWS offers only [Amazon EKS](/guides/eks-clusters) in the create wizard and the `aws` cluster commands and endpoints are not served. [Contact support](/platform/support) to have it turned on for your organisation. This is the self-managed lane. If you want AWS to run the control plane, use [Amazon EKS](/guides/eks-clusters) instead; the [comparison table](/guides/managed-kubernetes#managed-vs-self-managed) sets the two side by side. *** ## Prerequisites An AWS credential whose role carries the **Provisioning** scope - the one write role for EKS and self-managed clusters - or the EC2-only **Self-managed** scope; access keys with the equivalent policy also work. A provisioning stack launched before the self-managed statements were added must be updated. See [AWS Credentials](/platform/credentials/aws#provisioning). An SSH public key for node access. You can provide your own or let Ankra generate one. See [SSH Key Credentials](/platform/credentials/ssh-key). Nothing else needs to exist in the account beforehand: by default Ankra creates the network the cluster runs in. Only an [adopted VPC](#adopting-an-existing-vpc) asks you to prepare subnets first. ### The network Ankra creates Leave `vpc_id` out of the request (the wizard's **Create a new VPC (recommended)**) and Ankra lays out, in the region you chose: | Ankra creates | Detail | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A VPC** | From `network_ip_range` (default `10.0.0.0/16`, any `/16` to `/20`), with DNS support and DNS hostnames enabled. | | **Availability zones** | One zone when `control_plane_count` is below 3, three otherwise, picked from the region's available zones in name order - or the `availability_zones` you name. More than one zone needs at least three control planes, so etcd keeps quorum when a zone is lost. | | **Subnets** | Per zone, one private `/20` for the nodes and one public `/24`. Nodes never receive a public IP; the bastion lives in the public subnet of the first zone. | | **Internet gateway and route tables** | One internet gateway, a public route table with `0.0.0.0/0` on it, and one private route table per zone carrying that zone's egress route. | | **Egress** | By `egress_mode` - a NAT gateway per zone (the default) or the bastion as NAT; see [Egress modes](#egress-modes). | Preflight reports the plan before anything is built: `network_ownership: created`, `resolved_availability_zones`, whether the CIDR holds a `/20` and a `/24` per zone, whether every zone exists and is available, the three-control-plane rule, and the VPC, NAT gateway and Elastic IP quotas. Teardown deletes all of it - route tables, subnets, gateways, NAT gateways with their Elastic IPs, the VPC - by the tags Ankra set, in reverse order. ### Adopting an existing VPC An account with an established network passes `vpc_id` (the wizard's **Use an existing VPC**) and Ankra builds inside it instead. The target region then needs: | You provide | Requirements | | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **A VPC** | Any VPC in the region. Its DHCP options set, internet gateway and existing route tables are read, never changed. | | **One or more node subnets** (`node_subnet_ids`) | Private subnets for the control plane and workers. Spread them across availability zones for a multi-AZ cluster - Ankra places control planes one per zone and balances each node group across them. | | **One bastion subnet** (`bastion_subnet_id`) | A public subnet (a route to the internet gateway, and public IP assignment allowed) for the bastion. | | **Egress for the node subnets** | Nodes pull images, reach the Ankra platform and talk to AWS APIs, so their subnets need a default route somewhere - `existing` or `bastion_nat` below. | `network_ip_range`, `availability_zones` and `nat_gateway_single_zone` are refused alongside `vpc_id`; `node_subnet_ids` and `bastion_subnet_id` are refused without it. In an adopted VPC Ankra never creates, modifies or deletes your VPC, subnets, internet gateway, NAT gateways, DHCP options or existing route tables. ### Egress modes Nodes have no public IP, so every private subnet must route `0.0.0.0/0` through something. The `egress_mode` field on the create request decides what, and Ankra never mixes modes on one cluster: * **`nat_gateway`** (created VPC only, the default there) - an Ankra-created NAT gateway with its own Elastic IP in every zone, each zone's private route table pointing at its own gateway, so egress survives a zone outage. `nat_gateway_single_zone: true` creates one gateway in the first zone that every private subnet routes through - cheaper, one egress failure domain. **Cost:** about USD 32 per month per NAT gateway plus per-GB data processing (see [VPC pricing](https://aws.amazon.com/vpc/pricing/)); a three-zone cluster runs three gateways unless you collapse them to one. The bastion is an SSH jump host only. * **`bastion_nat`** (either network) - the bastion becomes the NAT instance, as it is on Hetzner and DigitalOcean: source/destination checking is switched off and it forwards node traffic to the internet gateway. In a created VPC the private route tables point their default route at the bastion; in an adopted VPC Ankra creates **one** tagged route table with `0.0.0.0/0` to the bastion and associates the node subnets with it, recording each subnet's previous association so teardown restores it exactly. **Cost:** the bastion instance and its Elastic IP, nothing else - the cheapest option. The trade is that node egress is capped by the bastion's network performance, shares its fate (a bastion resize interrupts it), and the bastion must stay running. * **`existing`** (adopted VPC only) - your node subnets already route the default to a NAT gateway, a NAT instance or a transit path. Ankra checks the route and changes nothing. **Cost:** whatever you already pay for that path, shared with everything else in the subnet. An omitted `egress_mode` resolves to `nat_gateway` for a created VPC. For an adopted VPC preflight resolves it: subnets that already have egress select `existing`; empty subnets without it select `bastion_nat`; subnets without egress that hold instances Ankra did not create are refused with a message naming them. `bastion_nat` is **always refused** in an adopted VPC when any node subnet carries a foreign instance, because re-pointing that subnet's default route would hijack its egress. Create the NAT gateway yourself and use `existing` in that case. ### What Ankra creates, and what it never touches Every object Ankra creates is tagged `ankra.cloud/cluster-id=`, `ankra.cloud/managed=true` and `kubernetes.io/cluster/=owned`, and the credential's write permissions are scoped to those tags. In both modes Ankra creates two security groups (nodes, bastion), an imported key pair, one Elastic IP for the bastion, the bastion instance, the node instances with their network interfaces and encrypted gp3 root volumes, and two IAM roles with instance profiles (`ankra-k3s--cp` and `ankra-k3s--node`). In a created VPC it also owns the VPC, subnets, internet gateway, route tables, NAT gateways and their Elastic IPs; in an adopted VPC it adds at most one route table (`bastion_nat` mode) and never touches your VPC, subnets, internet gateway, NAT gateways, DHCP options or existing route tables. The full list, with tags and IAM scoping, is in the [AWS Reference](/reference/aws#what-ankra-creates). *** ## Creating an AWS Cluster ### Via the Platform UI Go to **Clusters** in the Ankra dashboard and click **Create Cluster**. Choose **AWS** as the provider and pick the **Self-managed** action (the other action, **Cloud Managed**, creates [EKS](/guides/eks-clusters)). Pick an AWS credential with the provisioning or self-managed scope and an SSH key credential. A read-only (cost) credential, or a provisioning role whose stack predates the self-managed statements, is listed but refused at preflight with the missing permission named. You can create either credential from the wizard. Select a region - the list loads live from your credential. **Create a new VPC (recommended)** is selected: keep the network range (`10.0.0.0/16`) or set your own `/16` to `/20`, and optionally name the availability zones - otherwise preflight picks one zone for a single control plane and three for a three-node control plane. Switch to **Use an existing VPC** to pick the **VPC**, one or more **node subnets** and the **bastion subnet** instead; the wizard shows each subnet's availability zone and whether it has a public or private route, and offers only public subnets for the bastion. For a new VPC choose **NAT gateway** (the default - one per zone, or tick **single NAT gateway** to share one) or **bastion as NAT** (cheapest); for an existing VPC choose **existing**, **bastion as NAT**, or leave it on *Detect* and let preflight resolve it. Then enter the **allowed IPs** that may reach the bastion on port 22 - at least one CIDR, and `0.0.0.0/0` is refused. Your current public address is offered as a starting point. Set your cluster topology: * **Bastion** - instance type for the SSH bastion (a burstable type such as `t3.small` is enough for an SSH jump host; size it for bandwidth in `bastion_nat` mode) * **Control Plane** - count (1 to 9 - at least three when the cluster spans more than one availability zone, and three is what makes a new VPC three-zone) and instance type * **Workers** - one or more node groups, each with a name, instance type and count Instance types load live for the region with vCPUs, memory, architecture and hourly price. Only `amd64` types are offered in this release. Keep **k3s** (the wizard's default) or pick **kubeadm**, which on AWS is Cilium only. The CNI defaults to Cilium for both distributions; a k3s cluster can pick Calico or flannel and the [advanced features](/guides/digitalocean-clusters#advanced-cni-features), but only Cilium and Calico can run the [IMDS guard](#instance-metadata-imds) that keeps pods away from the control-plane instance role, so flannel is accepted with a preflight warning. The Ubuntu series defaults to 24.04 and the root volume is an encrypted gp3 volume of 40 GiB whose size you can raise (up to 2000 GiB). The wizard and the CLI examples on this page create k3s clusters. An API request that omits `distribution` follows the platform default, `kubeadm`, like every other self-managed provider - set `"distribution": "k3s"` explicitly when you want k3s from the API. Optionally connect a GitHub repository for GitOps-driven deployment. Two checkboxes on this step are on by default: * **Include Networking Stack** - deploys Traefik, cert-manager and a Let's Encrypt ClusterIssuer. Traefik's Service asks for a Network Load Balancer, which the AWS cloud controller creates in your account. * **Include Public DNS** - gives the cluster its own delegated subdomain on `ankra.cc` and installs external-dns with its credentials already wired, so an ingress hostname under that subdomain gets its DNS record and TLS certificate automatically. The wizard runs the [preflight](#preflight) before it submits and shows what it found - network ownership, the availability zones it resolved, the egress mode, the AMI it will use, and the vCPU, VPC, NAT gateway, Elastic IP and security-group quotas it checked. Click **Create** to start provisioning. A live progress view tracks IAM roles and instance profiles, **Prepare VPC, Subnets & Routing**, NAT gateways (in `nat_gateway` mode), security groups, key pair, Elastic IP, bastion, route table (`bastion_nat` in an adopted VPC), node instances, Kubernetes installation, cloud controller and CSI driver, and Ankra Agent setup. The cluster appears **offline** until provisioning completes, then transitions to **online**. ### Managing from the Dashboard Once the cluster is online, day-2 operations live where they do for every self-managed provider: * **Nodes** section - node groups, control plane, bastion health and resize, restart, diagnose and repair. See [Nodes](/platform/cluster-nodes). * **Settings → General** - Kubernetes upgrade, stop and start, [power schedules](/platform/cluster-power-schedules), and the **Danger Zone** to deprovision. * **Settings → Access** - SSH commands through the bastion and SSH key management. ### Via the CLI Create the credentials first. The AWS credential is the same one EKS uses: the provisioning role (`AnkraProvisioning`) builds both EKS and self-managed clusters, and the [credential page](/platform/credentials/aws#provisioning) walks through it in the dashboard, including how to update a stack launched before the self-managed statements were added. From the terminal, `onboarding` prints the external ID, the trust principal, the quick-create link for the stack and the `create-role` command to run once the stack shows `CREATE_COMPLETE`: ```bash theme={null} # 1. Print the external ID and the CloudFormation launch link for the provisioning role # (--scope self_managed instead for the EC2-only role that grants nothing for EKS) ankra credentials aws onboarding --scope provisioning # 2. Launch the stack from that link, wait for CREATE_COMPLETE, copy the RoleArn output, then store it ankra credentials aws create-role --name my-aws --role-arn --external-id \ --region eu-north-1 --scope provisioning # Access keys instead of a role: the secret is asked for on a masked prompt, never passed as a flag ankra credentials aws create-keys --name my-aws-keys --access-key-id --region eu-north-1 ankra credentials aws list # SSH key credential ankra credentials hetzner ssh-key create --name my-ssh-key --generate # provider-neutral; any SSH key credential works ``` The external ID `create-role` stores must be the one `onboarding` printed and the stack was launched with; a role created at the `cost` scope is refused at cluster create. Then look up what the credential can reach (`vpcs` and `subnets` matter only when you adopt a VPC): ```bash theme={null} ankra cluster aws regions --credential-id ankra cluster aws availability-zones --credential-id --region eu-north-1 ankra cluster aws vpcs --credential-id --region eu-north-1 ankra cluster aws subnets --credential-id --region eu-north-1 --vpc-id ankra cluster aws instance-types --credential-id --region eu-north-1 ankra cluster aws images --credential-id --region eu-north-1 ankra cluster aws pricing --credential-id --region eu-north-1 --instance-type t3.large ``` #### Preflight `preflight` takes the same flags as `create` and runs every check the create runs, without creating anything. For a created network: whether `network_ip_range` holds a private `/20` and a public `/24` per zone, whether every zone exists and is available in the region, the three-control-plane rule for more than one zone, and the VPC (`L-F678F1CE`), NAT gateway (`L-FE5A380F`) and Elastic IP quotas. For an adopted VPC: whether the VPC and subnets exist and sit in the region, which zones the node subnets cover, and the resolved egress mode (and why `bastion_nat` is refused, if it is). For both: the AMI resolved for the Ubuntu series and architecture, the vCPU quota for the chosen instance families, the security-groups-per-interface quota, and whether `bastion_allowed_ips` is valid. The result names `network_ownership` (`created` or `adopted`) and `resolved_availability_zones`. Every item is three-state - `ok`, `warning` (a read Ankra could not perform, or a caveat such as flannel not enforcing the IMDS guard) or `error` - and the result carries `can_proceed` and `resolved_egress_mode`, the mode the create will run under. Run it until there are no errors, then run `create` with the same flags. Only `--name`, `--credential-id`, `--ssh-key-credential-id`, `--region` and `--bastion-allowed-ips` are required; without `--vpc-id` Ankra creates the network. Three control planes make it a three-zone VPC with a NAT gateway per zone: ```bash theme={null} ankra cluster aws preflight \ --name my-cluster \ --credential-id \ --ssh-key-credential-id \ --region eu-north-1 \ --bastion-allowed-ips 203.0.113.10/32 \ --control-plane-count 3 \ --control-plane-type t3.medium \ --worker-count 3 \ --worker-type t3.large ``` #### Create ```bash theme={null} ankra cluster aws create \ --name my-cluster \ --credential-id \ --ssh-key-credential-id \ --region eu-north-1 \ --bastion-allowed-ips 203.0.113.10/32 \ --control-plane-count 3 \ --control-plane-type t3.medium \ --worker-count 3 \ --worker-type t3.large \ --distribution k3s \ --retention-policy retain ``` The created-network flags are `--network-ip-range` (default `10.0.0.0/16`), `--availability-zones` (comma-separated, in the order the spread walks them), `--egress-mode nat_gateway|bastion_nat` and `--nat-gateway-single-zone`. A single-zone development cluster on the cheapest egress: ```bash theme={null} ankra cluster aws create \ --name dev-cluster \ --credential-id \ --ssh-key-credential-id \ --region eu-north-1 \ --bastion-allowed-ips 203.0.113.10/32 \ --egress-mode bastion_nat \ --control-plane-count 1 \ --worker-count 2 \ --distribution k3s ``` #### Adopting an existing VPC from the CLI Pass `--vpc-id` with the subnets, and `--egress-mode existing|bastion_nat` (or leave it for preflight to detect); the created-network flags are refused alongside it: ```bash theme={null} ankra cluster aws create \ --name my-cluster \ --credential-id \ --ssh-key-credential-id \ --region eu-north-1 \ --vpc-id \ --node-subnet-ids ,, \ --bastion-subnet-id \ --egress-mode existing \ --bastion-allowed-ips 203.0.113.10/32 \ --control-plane-count 3 \ --worker-count 3 \ --distribution k3s ``` `--cni` and `--cni-features` pick the CNI and its feature toggles (`kube_proxy_replacement`, `hubble`, `wireguard_encryption` for Cilium, `ebpf_dataplane` for Calico); `--environment` and `--criticality` classify the cluster as on every provider; `--gitops-repository`, `--gitops-credential-name` and `--gitops-branch` wire GitOps at create. The full flag list is in the [CLI reference](/reference/cli/cluster). ### Via the API Preflight first - the endpoint takes the same body as create and answers with findings instead of a cluster. Without `vpc_id` the request describes a created network (`network_ownership: created` in the result, with the `resolved_availability_zones`): ```bash theme={null} curl -X POST https://platform.ankra.app/api/v1/clusters/aws/preflight \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "my-cluster", "credential_id": "", "ssh_key_credential_id": "", "region": "eu-north-1", "bastion_allowed_ips": ["203.0.113.10/32"], "control_plane_count": 3, "node_groups": [ {"name": "default", "instance_type": "t3.large", "count": 3} ] }' ``` Then create. The response is `{"cluster_id", "name", "kind": "aws", "state": "creating", "operation_id"}`; the operation tracks provisioning. ```bash theme={null} curl -X POST https://platform.ankra.app/api/v1/clusters/aws \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "my-cluster", "credential_id": "", "ssh_key_credential_id": "", "region": "eu-north-1", "network_ip_range": "10.0.0.0/16", "availability_zones": ["eu-north-1a", "eu-north-1b", "eu-north-1c"], "egress_mode": "nat_gateway", "nat_gateway_single_zone": false, "bastion_allowed_ips": ["203.0.113.10/32"], "control_plane_count": 3, "control_plane_type": "t3.medium", "node_groups": [ {"name": "default", "instance_type": "t3.large", "count": 3} ], "distribution": "k3s", "ubuntu_series": "24.04", "root_volume_gib": 40, "retention_policy": "retain" }' ``` #### Adopting an existing VPC from the API Name the VPC and its subnets instead; `network_ip_range`, `availability_zones` and `nat_gateway_single_zone` are refused with `422` alongside `vpc_id`, and `egress_mode` is `existing` or `bastion_nat`: ```bash theme={null} curl -X POST https://platform.ankra.app/api/v1/clusters/aws \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "my-cluster", "credential_id": "", "ssh_key_credential_id": "", "region": "eu-north-1", "vpc_id": "", "node_subnet_ids": ["", "", ""], "bastion_subnet_id": "", "egress_mode": "existing", "bastion_allowed_ips": ["203.0.113.10/32"], "control_plane_count": 3, "node_groups": [ {"name": "default", "instance_type": "t3.large", "count": 3} ], "distribution": "k3s" }' ``` Every request field, the defaults, the catalogs and the limits are in the [AWS Reference](/reference/aws). You can also ask Ankra's AI to create the cluster in chat - the `create_aws_cluster` [tool](/platform/mcp-tools#provisioning--nodes) takes the same fields and runs the preflight for you. *** ## Accessing the Cluster The bastion is the only instance with a public address, and its security group admits SSH only from `bastion_allowed_ips`. The nodes admit SSH and the Kubernetes API from the bastion's security group, and everything from each other. **Settings → Access** shows copy-pasteable commands with the addresses filled in, and `ankra cluster aws access-info ` (`GET /api/v1/clusters/aws/{cluster_id}/access-info`) answers `bastion_host`, `bastion_port` (22), `bastion_user` and `target_user` (both `ubuntu`) and the control plane IPs: ```bash theme={null} # SSH to a control plane node through the bastion ssh -J ubuntu@ ubuntu@ # Port-forward the Kubernetes API for local kubectl ssh -L 6443::6443 -N -J ubuntu@ ubuntu@ ``` The Ubuntu images log in as `ubuntu`. For everyday `kubectl` you do not need the tunnel - the [Kubernetes browser](/platform/kubernetes-workloads), `ankra cluster get` and the [kubeconfig](/guides/kubeconfig) route through the Ankra Agent, which needs no inbound access at all. ### Changing who may reach the bastion `bastion_allowed_ips` is the bastion security group's SSH ingress. Update it from **Settings → Access** or the API and the rule is rewritten in place; the list must keep at least one entry and never widen to `0.0.0.0/0`. Ankra's own provisioning path is always admitted. ### Managing SSH keys Add or remove SSH key credentials on a running cluster from **Settings → Access**, or with `ankra cluster ssh-keys set --ssh-key-credential-ids ,`. Changes are written to `authorized_keys` on every node on the next reconciliation; the key pair imported into EC2 at create is the Ankra-managed key and is never removed. `ankra cluster ssh-keys resync ` forces the write. *** ## Node Groups Node groups organise workers into groups with independent instance types, counts, labels and taints, exactly as on the other self-managed providers. New nodes of a group are balanced across the cluster's node subnets - each node takes the availability zone with the fewest instances cluster-wide - unless the group is pinned to one zone with `availability_zone`. Pin a group that runs zonal storage: an EBS volume cannot attach from another availability zone. ```bash CLI theme={null} ankra cluster node-group list ankra cluster node-group add --name workers-large --instance-type m6i.xlarge --count 2 ankra cluster node-group add --name database --instance-type r6i.large --count 1 --availability-zone eu-north-1a ankra cluster node-group scale default 4 ankra cluster node-group upgrade default m6i.large ankra cluster node-group delete workers-large ``` ```bash cURL theme={null} curl https://platform.ankra.app/api/v1/clusters/aws//node-groups \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST https://platform.ankra.app/api/v1/clusters/aws//node-groups \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "workers-large", "instance_type": "m6i.xlarge", "count": 2, "labels": {"tier": "backend"}, "taints": [{"key": "dedicated", "value": "backend", "effect": "NoSchedule"}] }' curl -X PUT https://platform.ankra.app/api/v1/clusters/aws//node-groups/default/scale \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"count": 4}' ``` An instance type change replaces every node of the group one at a time - the node is drained, terminated and recreated on the new type, because EC2 cannot resize an instance in place without a stop. Labels, taints and [autoscaling](/guides/cluster-autoscaling) bounds are edited with the same endpoints as every other provider; the full list is in the [Node Group API](/reference/aws#node-group-api-reference). Every node carries `topology.kubernetes.io/zone` and `topology.kubernetes.io/region`, set by the cloud controller, so `topologySpreadConstraints` work without extra configuration. *** ## Control Plane Grow a single control plane to three, or change the control plane instance type, from the **Nodes → Control plane** tab or the CLI. Three control planes are required when the node subnets span more than one availability zone, so that etcd keeps quorum when a zone is lost; a multi-AZ cluster cannot be scaled below three. ```bash theme={null} ankra cluster aws control-plane get ankra cluster aws control-plane set-count 3 ankra cluster aws control-plane set-instance-type t3.large # cluster must be stopped ``` *** ## Restarting, Diagnosing and Repairing a Node Restart any node - a control plane node, a worker, or the bastion - as a tracked operation from the **Nodes** section, the CLI or the API. A restart is an EC2 reboot; if the instance does not respond it falls back to a stop and start, which keeps the private IP. ```bash CLI theme={null} ankra cluster aws nodes list ankra cluster aws nodes restart ankra cluster aws nodes cloud-init-log # cloud-init status and the tail of its log, over the bastion ``` ```bash cURL theme={null} curl https://platform.ankra.app/api/v1/clusters/aws//nodes \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST https://platform.ankra.app/api/v1/clusters/aws//nodes//restart \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST https://platform.ankra.app/api/v1/clusters/aws//nodes//cloud-init-log \ -H "Authorization: Bearer $ANKRA_API_TOKEN" ``` See [Restarting a Node](/guides/hetzner-clusters#restarting-a-node) for the response shape and state requirements - identical across providers. You can also ask the AI assistant ("restart worker-2 on my-cluster"). Diagnose and repair (an OS-level check over SSH through the bastion, and a repair of the node's k3s or kubelet unit) run from the node's row menu in the **Nodes** section or through the `ssh_diagnose_node` and `ssh_repair_node` [AI tools](/platform/mcp-tools#provisioning--nodes). ### Bastion health and resize `bastion status` reports the last verdict of the bastion health loop - reachable or not, which hop a failed probe stopped at, and when - without probing anything, so it answers even while the bastion is down; `bastion diagnose` probes it now as a tracked operation. Both are shown on the **Nodes → Bastion** tab. ```bash CLI theme={null} ankra cluster aws bastion status ankra cluster aws bastion diagnose ``` ```bash cURL theme={null} curl https://platform.ankra.app/api/v1/clusters/aws//bastion/health \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST https://platform.ankra.app/api/v1/clusters/aws//bastion/diagnose \ -H "Authorization: Bearer $ANKRA_API_TOKEN" ``` Resize the bastion from the **Nodes → Bastion** tab or with the API - there is no CLI verb for it on AWS - `PUT /api/v1/clusters/aws/{cluster_id}/bastion/instance-type` with `{"instance_type": "t3.medium"}`: the instance is stopped, its type changed and started again, keeping its Elastic IP. In `bastion_nat` mode the bastion is the nodes' default route, so a resize or restart interrupts every node's outbound traffic - image pulls, the Ankra Agent connection, AWS API calls from the cloud controller - until it is back. *** ## Upgrading Kubernetes Version Upgrades roll control planes first, then workers, one node at a time, each cordoned, drained and gated on being `Ready` at the target version. Both k3s and kubeadm clusters upgrade; downgrades and skipping a minor version are refused. The AWS cloud controller manager's minor version is pinned to the cluster's Kubernetes minor, and the upgrade path moves it with the nodes. ```bash CLI theme={null} ankra cluster aws k8s-version ankra cluster k3s-versions ankra cluster upgrade v1.35.1+k3s1 ``` ```bash cURL theme={null} curl https://platform.ankra.app/api/v1/clusters/aws//k8s-version \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST https://platform.ankra.app/api/v1/clusters/aws//upgrade-k8s-version \ -H "Authorization: Bearer $ANKRA_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"target_version": "v1.35.1+k3s1"}' ``` *** ## Stopping and Starting a Cluster Stop a cluster to park it: every node and the bastion are stopped (not terminated), so the instances keep their root volumes and - because they live in a VPC - their private IPs. Start brings the same instances back and reconciles the cluster; embedded etcd rejoins on the same addresses, so a stopped cluster comes back with its state, not as a rebuild. The Elastic IP stays allocated and the bastion keeps its public address across a stop. ```bash CLI theme={null} ankra cluster aws stop ankra cluster aws start # scope defaults to "all" ankra cluster aws start --scope control_plane # control plane only ``` ```bash cURL theme={null} curl -X POST https://platform.ankra.app/api/v1/clusters/aws//stop \ -H "Authorization: Bearer $ANKRA_API_TOKEN" curl -X POST "https://platform.ankra.app/api/v1/clusters/aws//start?scope=all" \ -H "Authorization: Bearer $ANKRA_API_TOKEN" ``` While stopped you pay for EBS storage (root volumes and any volumes the CSI driver provisioned), the bastion's Elastic IP, any load balancers the cloud controller created, and - in a created VPC - the network itself: a stop keeps the VPC and its NAT gateways, which keep billing by the hour, so a cluster parked for weeks on `nat_gateway` egress is cheaper to deprovision. Compute is not billed. Ankra records each instance's private IP at create and verifies it after every start; a difference is reported as drift rather than silently accepted. Stop and start also run on a timetable with [power schedules](/platform/cluster-power-schedules), and a stopped cluster is where control plane instance type changes are applied. *** ## Deprovisioning Deprovisioning terminates the instances, releases the bastion's Elastic IP, and deletes the security groups, key pair and per-cluster IAM roles. In a created VPC it then removes the network in reverse order - NAT gateways with their Elastic IPs, route tables, subnets, the internet gateway, the VPC - every object found by the tags Ankra set. In an adopted VPC it restores each node subnet's previous route table association before deleting the one route table it created (`bastion_nat` mode), and your VPC, subnets, gateways and DHCP options are untouched. The cluster is removed from Ankra. This action is irreversible. All data on the nodes' root volumes is permanently deleted. What happens to the volumes the EBS CSI driver provisioned and the load balancers the cloud controller created follows the cluster's `retention_policy`, chosen at create (with one exception: when Ankra created the VPC, load balancers are always deleted with it, because a load balancer cannot outlive the VPC it lives in - volumes still follow the policy): * **`retain`** (the default) records and keeps them. EBS volumes and Network Load Balancers stay in your account and keep billing; the teardown result lists their provider IDs, and they are tagged with the cluster ID so they are easy to find in the EC2 and load balancing consoles. * **`delete`** sweeps them: a volume or load balancer is deleted only when it carries this cluster's `ankra.cloud/managed=true` tag **and** the policy is `delete`. Nothing without the tag is touched, whatever the policy. ```bash CLI theme={null} ankra cluster aws deprovision ``` ```bash cURL theme={null} curl -X DELETE https://platform.ankra.app/api/v1/clusters/aws/ \ -H "Authorization: Bearer $ANKRA_API_TOKEN" ``` The same sweep applies to a cluster that was stopped earlier: the volumes and load balancers recorded at stop time are still known and follow the policy. *** ## Architecture An AWS cluster provisions the following infrastructure - the first four rows only when Ankra creates the network: | Component | Description | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **VPC** | Created from `network_ip_range` with DNS support and hostnames, and an internet gateway | | **Subnets** | Per availability zone (one or three), a private `/20` for the nodes and a public `/24` for the bastion and the NAT gateway | | **Route tables** | One public table with `0.0.0.0/0` on the internet gateway, and one private table per zone carrying that zone's egress route | | **NAT gateways** | `nat_gateway` mode: one per zone with its own Elastic IP, or one for all zones with `nat_gateway_single_zone` | | **Security groups** | One for the nodes (all traffic within the group, SSH and Kubernetes API from the bastion group, load balancer health checks added by the cloud controller) and one for the bastion (SSH from `bastion_allowed_ips` only) | | **Key pair** | Your SSH key credential's public key, imported as `ankra-k3s-` | | **Elastic IP** | One, attached to the bastion - the cluster's only public address | | **Bastion** | EC2 instance in the public subnet - SSH jump host Ankra uses to provision and manage nodes, and the NAT instance in `bastion_nat` mode | | **Control plane(s)** | EC2 instances in the node subnets, no public IP, one per availability zone on a multi-AZ cluster | | **Worker(s)** | EC2 instances in the node subnets, no public IP, organised in [node groups](#node-groups) | | **IAM roles and instance profiles** | `ankra-k3s--cp` (cloud controller manager and EBS CSI controller) and `ankra-k3s--node` (kubelet and CSI node) | | **Route table (adopted VPC)** | `bastion_nat` mode only: one route table with `0.0.0.0/0` to the bastion, associated with your node subnets | | **Cloud provider stack** | The AWS cloud controller manager and the EBS CSI driver, deployed after the Ankra Agent | ```mermaid theme={null} flowchart TB Internet((Internet)) Platform[Ankra Platform] subgraph VPC [VPC - created by Ankra, or adopted and never modified] subgraph Public [Public subnets - one per zone] Bastion[Bastion
Elastic IP + private IP] end subgraph Private [Private subnets - one per zone] CP[Control plane instances
private IP only] Workers[Worker instances
private IP only] Agent[Ankra Agent
runs in the cluster] end IGW[Internet gateway] NAT[NAT gateway per zone - nat_gateway mode
the bastion - bastion_nat mode
your own path - existing mode] end NLB[Network Load Balancer
created by the AWS cloud controller] Platform -->|SSH from allowed IPs only| Bastion Bastion -->|SSH jump| CP Bastion -->|SSH jump| Workers Agent -->|outbound only - NATS| Platform CP -->|default route| NAT Workers -->|default route| NAT NAT --> IGW Bastion --> IGW IGW --> Internet Internet --> NLB NLB --> Workers ``` ### Cloud integration * **Cloud controller manager.** Ankra deploys the AWS cloud controller manager as a vendored manifest with an exact image tag, matched to the cluster's Kubernetes minor. Nodes register with `cloud-provider=external`, a provider ID of `aws:////`, and their EC2 private DNS name as the node name, so the controller can find each instance. `KubernetesClusterID` is the Ankra cluster ID, and every instance carries the matching `kubernetes.io/cluster/=owned` tag. * **Load balancers.** A `Service` of type `LoadBalancer` gets an AWS load balancer from the cloud controller. The Traefik the networking stack installs asks for a Network Load Balancer (`service.beta.kubernetes.io/aws-load-balancer-type: nlb`), and Ankra sets the managed tag on it so the teardown sweep can find it. * **Storage.** The EBS CSI driver is installed from its upstream chart at a pinned version, after the cloud controller (it needs the provider ID and topology labels the controller sets). The default StorageClass provisions **encrypted gp3** volumes. * **Credentials without keys.** Neither the cloud controller nor the CSI driver holds AWS keys: each uses the instance profile of the node it runs on. The control-plane and worker profiles are distinct, following the upstream policy split, and the platform credential you gave Ankra is never copied into the cluster. * **Instance metadata.** Every instance requires IMDSv2. Workers and the bastion use a hop limit of 1, so a pod on the pod network cannot reach the instance role and a workload does not inherit the node's AWS permissions by accident. Control-plane instances use a hop limit of **3**, because the upstream EBS CSI chart cannot run its controller on the host network and that controller needs the control-plane role. The compensating control is the **IMDS guard** shipped with the AWS cloud-provider stack: a cluster-wide network policy, rendered for Cilium and Calico, that denies pod egress to `169.254.169.254` except from the EBS CSI controller. On a flannel cluster there is no policy engine to enforce it - the create result reports `imds_guard: unavailable`, and a pod scheduled onto a control-plane node can reach the control-plane instance role. Prefer Cilium (the default) or keep the control planes tainted. A workload that needs AWS access brings its own credentials either way. *** ## Troubleshooting ### Common Issues | Issue | Solution | | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Preflight refuses the credential | The credential is read-only (cost scope), or its provisioning stack predates the self-managed statements. Add a [provisioning credential](/platform/credentials/aws#provisioning), or update the existing stack to the published template | | The preflight `cidr` item is an error | `network_ip_range` does not hold a private `/20` and a public `/24` per zone - a `/20` VPC fits one zone only. Widen the range (up to `/16`) or use fewer zones | | Preflight refuses the zones | A named zone does not exist or is not available in the region, or more than one zone was given with fewer than three control planes. Check `ankra cluster aws availability-zones`; set `control_plane_count` to 3 | | Preflight reports the VPC or NAT gateway quota | The account is at its VPCs-per-region (`L-F678F1CE`) or NAT-gateways-per-zone (`L-FE5A380F`) limit. Request an increase, deprovision unused clusters, or use `bastion_nat` / `nat_gateway_single_zone` to need fewer gateways | | Preflight refuses `bastion_nat` (adopted VPC) | A node subnet holds an instance Ankra did not create. Move it, pick an empty subnet, or create a NAT gateway and use `existing` | | Preflight reports no egress and refuses `existing` (adopted VPC) | The node subnets' route table has no `0.0.0.0/0` route to a NAT gateway, NAT instance or transit path. Add one, or use empty subnets with `bastion_nat` | | `node_subnet_ids` or `network_ip_range` refused with `422` | The subnet fields belong to an adopted VPC and the network fields to a created one - a request carries `vpc_id` with subnets, or neither | | `bastion_allowed_ips` refused | The list is empty or contains `0.0.0.0/0`. Give it at least one CIDR | | Cluster stuck in provisioning | Check the vCPU quota for the instance family, the Elastic IP and NAT gateway quotas, and - for an adopted VPC - that the bastion subnet is public (route to the internet gateway) | | Nodes stay `NotReady` with the `node.cloudprovider.kubernetes.io/uninitialized` taint | The cloud controller cannot match the node - see [cluster ID and node names](#cloud-controller-cluster-id-and-node-names) below | | `LoadBalancer` Service stuck in `Pending` | Check the cloud controller pods in `kube-system`; on a fresh account the first load balancer also creates the Elastic Load Balancing service-linked role, which the credential allows | | PVC stuck in `Pending` | Check the EBS CSI controller pods; a pod pinned to a zone with no volume there needs a node group in that zone | | Cannot SSH to the bastion | Your address is not in `bastion_allowed_ips`, or the security group was edited outside Ankra. Update the list from **Settings → Access** | | Nodes lost egress after a bastion resize | Expected in `bastion_nat` mode while the bastion restarts; it recovers when the bastion is back | | Create result reports `imds_guard: unavailable` | The cluster runs flannel, which cannot enforce the IMDS guard. Recreate with Cilium or Calico, or keep the control planes tainted - see [Instance metadata](#instance-metadata-imds) | ### Cloud controller cluster ID and node names The AWS cloud controller manager identifies a cluster's instances by the `kubernetes.io/cluster/=owned` tag and matches each Kubernetes node to an instance by name. Two things must therefore hold on every node, and Ankra sets both: * The instance carries the `kubernetes.io/cluster/` tag, where `` is the cluster ID the controller was configured with. Removing or editing that tag in the console makes the controller treat the instance as foreign; the node keeps the `uninitialized` taint (or, on a running cluster, is removed from Kubernetes) until the tag is restored. * The node name equals the instance's EC2 private DNS name, as reported by `DescribeInstances`. k3s is started with `--node-name` set to that value, so the name does not depend on what the OS derives from the VPC's **DHCP options set**. A VPC whose DHCP options carry a custom `domain-name` produces hostnames such as `ip-10-0-1-23.corp.example`, while EC2 reports `ip-10-0-1-23.eu-north-1.compute.internal`; the controller resolves the latter, which is why Ankra pins the node name rather than the hostname. Changing the DHCP options set after creation does not affect existing or new nodes. If a node was renamed or re-registered by hand and no longer matches, delete the Kubernetes node object and restart the node from the **Nodes** section; it registers again under the correct name. ### Instance metadata (IMDS) Every instance is launched with IMDSv2 required, so tools that use IMDSv1 (`curl http://169.254.169.254/latest/...` without a token) fail on the nodes. Workers and the bastion have a hop limit of 1, so a pod on the pod network fails on either version - that is the intended isolation of the instance role. Control-plane instances have a hop limit of 2 so the EBS CSI controller, which the upstream chart cannot run on the host network, can use the control-plane role; the **IMDS guard** - a cluster-wide network policy in the AWS cloud-provider stack, rendered for Cilium and Calico - denies every other pod's egress to `169.254.169.254`. The guard needs a policy engine. On a flannel cluster the create result reports `imds_guard: unavailable`, and any pod scheduled onto a control-plane node can read the control-plane instance role's credentials. Prefer Cilium (the default) or Calico, or keep the control planes tainted so no workload lands there. If a workload needs AWS credentials, give it its own (a secret, or a role it assumes with keys you manage); do not raise the hop limits or switch IMDSv1 back on, and note that `ModifyInstanceMetadataOptions` is one of the calls Ankra's credential is allowed to make on its own instances, so Ankra restores the hardening on reconciliation. ### AWS service quotas Preflight checks the quotas that most often block a create, but a quota can still be consumed between preflight and create, and the per-cluster quota report (`GET /api/v1/clusters/aws/{cluster_id}/quotas`) answers `404` (`Quota reporting is not available for aws clusters`) until it ships. If provisioning fails with a quota error, check in the [Service Quotas console](https://console.aws.amazon.com/servicequotas/) for the region: * Running On-Demand instances (vCPU) for the instance family * VPCs per region (`L-F678F1CE`) - a created network uses one * NAT gateways per availability zone (`L-FE5A380F`) - `nat_gateway` egress uses one per zone, or one in total with `nat_gateway_single_zone` * Elastic IPs per region - one for the bastion plus one per NAT gateway * Security groups per network interface, and rules per security group * Network Load Balancers per region Request an increase, then create again - a failed create leaves nothing behind but tagged objects Ankra removes on the next attempt.