Skip to main content
This use case is the shape of a real estate Ankra operates: one infrastructure cluster that every other cluster leans on, and a production and a development cluster that run the applications. Everything lives in one GitOps repository, every secret is sealed with SOPS, every stateful stack is protected into a backup vault, and every alert reaches one on-call channel. It is long because it is complete. Each step is one stack in one cluster file, so you can take the parts you need and leave the rest.

​
Build it with your AI coding assistant

This guide is written so an assistant that has the Ankra CLI and Ankra’s agent skills can carry it out for you, one step at a time, while you review. The example below uses Claude Code; the same steps work for Cursor, Codex, Copilot, Windsurf and Gemini CLI, see Agent Skills.
1

Install the CLI and sign in

Pick the organisation the clusters will live in. The CLI page has the installer for machines without Homebrew.
2

Give the assistant the Ankra skills

This installs the ankra-* skills - platform principles, cloud clusters, stacks and add-ons, SOPS, backups, alerts, observability, domains and DNS, security - so the assistant knows the CLI, the YAML shapes and the order of operations in this guide. The hook pauses it before destructive commands and keeps it on ankra instead of kubectl or helm.
3

Hand it this page

Start the assistant in the GitOps repository and give it the page and your inputs. A prompt that works:
Smaller asks work too: “Do Step 4 only, the central metrics stack” or “Protect every stack with data on the production cluster into production-backups”.
4

Review, then let it continue

Each step ends with something you can check: a draft to open in the portal, an ankra cluster operations list that reads up, a curl that returns the right status. Approve the draft, tell the assistant to go on, and finish with the Verify list at the end of this page.Two things it must never do on its own, and the skills tell it so: apply to the production cluster without a draft you have seen, and commit a secret in plaintext.
The production and development clusters send metrics by remote_write and logs by _bulk into the infrastructure cluster's ingest endpoints and restore points into a backup vault per environment. Inside the infrastructure cluster, Prometheus with Alertmanager and OpenSearch with Dashboards receive the data, Grafana reads Prometheus and Tempo, Harbor and OneUptime sit beside them, and a staff sign-in realm on Kratos, Hydra and oauth2-proxy signs people into all four tools. Alertmanager and Ankra notifications deliver to one on-call channel.

Hub and spokes: the infrastructure cluster receives, the application clusters send

​
What you build

Nothing here is specific to one cloud. The example creates the clusters on UpCloud because that is where this layout runs today; swap the create command for Hetzner, Scaleway or any other Ankra Managed provider and the stacks are unchanged, apart from the storage class names.

​
Prerequisites

  • An Ankra organisation where you are an admin, and the CLI logged in.
  • A cloud credential and an SSH key credential for the provider you build on (UpCloud credentials).
  • A Git credential with write access to the GitOps repository (Git credentials).
  • A domain you control, for example example.com, and a DNS credential Ankra can publish records with (custom DNS zones covers the Avura route; ankra org dns credentials create stores an external-dns webhook credential for any other provider).
  • A Microsoft 365 (Entra ID) tenant for staff sign-in. Any OpenID Connect provider Kratos supports works the same way.
  • A Microsoft Teams or Slack webhook for the on-call channel.
  • The charts come from registries Ankra already ships in its catalogue for every organisation: prometheus-community, grafana, opensearch-k8s-operator, fluent, oauth2-proxy, cloudnative-pg, harbor and oneuptime. The one exception is Ory’s chart repository for Kratos and Hydra, connected once:
Write the cluster files by hand in the layout below and apply them with ankra cluster apply, or build the stacks in the portal and let Ankra write them to Git. Both end in the same repository. This page shows the files, because they are what you review, diff and roll back.

​
Step 1: Create the clusters

Three clusters, created with the GitOps flags so Ankra commits their generated stacks to your repository from the first minute. The control plane count is the one choice you cannot change later: three for anything that must survive losing a node.
1

Store the credentials

2

Create the infrastructure cluster

Three control planes, three workers. The OpenSearch nodes, Prometheus and Harbor all want memory, so size the workers for them.
--include-networking and --include-dns are on by default: the cluster arrives with Traefik behind a cloud load balancer, cert-manager, a letsencrypt-prod ClusterIssuer and external-dns for its generated subdomain under ankra.cc.
3

Create the production cluster

Same shape, sized for the applications. Keep the control plane at three.
4

Create the development cluster

One control plane is fine here: losing it loses nothing you cannot rebuild from Git.
5

Wait for online, then look at the repository

create submits and returns. Watch the cluster list until all three read online, then pull the repository. Ankra has written one directory per cluster:
Every stack you add below is another entry under spec.stacks in that cluster’s YAML, with its values and manifests next to it. See GitOps for how Ankra and your commits share the directory.
A control plane count, the CNI, the etcd topology and the network mode are fixed at creation. Everything else - node groups, plans, Kubernetes version, autoscaling - changes later. See Choices fixed at create time.

​
Step 2: Organisation-wide DNS, alerting and vaults

These are declared once and inherited by every cluster, including clusters you create next year.
1

Serve your own zone from every cluster

The external-dns Ankra installs publishes only under the generated ankra.cc subdomain. For example.com, store a DNS credential and declare the zone for the organisation. Ankra renders a second external-dns per cluster, scoped to exactly that zone, with its own record ownership.
From here on, an Ingress with external-dns.alpha.kubernetes.io/hostname: metrics.example.com gets its record, and cert-manager.io/cluster-issuer: letsencrypt-prod gets its certificate.
2

One destination, two routes

Ankra’s own notifications - a failed deployment, a cluster whose agent went offline, a GitOps sync that stopped, new severe CVEs - go to the same channel as Alertmanager will.
ankra alerts routes preview --kind gitops_sync_failed --severity warning --cluster-id <cluster-id> shows which destinations a notification would reach before one happens. Notification routing explains the evaluation order.
3

One backup vault per environment

A vault is an S3-compatible bucket Ankra verifies and keeps the keys for. Let Ankra create the buckets on the same provider credential; keep production and development apart so a retention mistake in one cannot touch the other.
Both should read ready. Nothing is backed up yet; Step 9 attaches stacks to them.

​
Step 3: Infrastructure cluster - storage and secrets

Two small stacks that everything else depends on. They go into clusters/infrastructure-<short-id>/infrastructure.yaml under spec.stacks.
1

A storage class that keeps its disks

The provider’s default StorageClass reclaims a volume the moment its claim is deleted. Prometheus, OpenSearch, Harbor and the identity database all get a class with reclaimPolicy: Retain, and a snapshot class the backup data plane can use.
On another provider, change provisioner, driver and parameters to that CSI driver’s values.
2

Pre-created Secrets, sealed

The passwords Grafana, OpenSearch, the log shippers and Harbor read are created in one stack that the consumers list as a parent, so a Secret always exists before the chart that mounts it. Write them in plaintext, then seal them before committing.
Seal every Secret in file mode before the commit; the CLI rewrites the file in place and verifies each listed value is ENC[...]:
--all-data seals every key under data and stringData of a Secret. For a value inside Helm values, ankra cluster encrypt addon <name> --key <key> -f <cluster.yaml> does the same. SOPS Encryption covers rotation and glob keys.
Ankra identifies a manifest by its name. Renaming the manifest that holds a namespace, a PersistentVolumeClaim or a database deletes those objects and creates new ones. Keep the names of stateful manifests fixed from the first commit. See Protect Persistent Data.

​
Step 4: Infrastructure cluster - central metrics with Prometheus, Grafana and Tempo

One Prometheus holds every cluster’s series. The hub scrapes itself; the spokes remote-write into it through a public endpoint that accepts exactly one path from exactly their addresses. Grafana queries only this Prometheus, so a dashboard with a cluster drop-down shows the whole estate.
1

kube-prometheus-stack as the receiver

Three lines do the multi-cluster work. enableRemoteWriteReceiver opens /api/v1/write. The default scrapeClass stamps cluster=infrastructure on the hub’s own scrapes so they sit next to the spokes’ series, which carry their own cluster external label. group_by includes cluster so one noisy spoke does not fold into another’s alert. Every rule in this Prometheus now evaluates the spokes’ series too, so their alerts reach the channel without any Alertmanager on the spokes.Seal the webhook before committing: ankra cluster encrypt addon kube-prometheus-stack --key webhook_url -f <cluster.yaml>. Alertmanager is the one place a Slack receiver is slack_configs with api_url instead; everything else is identical.
2

Grafana with Prometheus and Tempo data sources

Grafana is a separate add-on so it can be upgraded, restarted and gated independently of Prometheus. The sign-in block refers to the gate from Step 6; leave auth.proxy out until that stack exists, and the local admin from grafana-admin is the only account.
3

Tempo for traces

Tempo’s metrics generator writes service-graph and span metrics back into the same Prometheus, which is what makes the Grafana service map work across clusters.
Applications send OTLP to tempo.tempo.svc.cluster.local:4317 from inside the hub, or through an Ingress on traces-ingest.example.com built the same way as the metrics endpoint below.
4

The metrics-ingest endpoint

This is the only way into the hub’s Prometheus from outside the cluster: one exact path, TLS from Let’s Encrypt, and an IP allow-list of the spokes’ egress addresses. No credential is needed because the endpoint can only append samples, never read them.
The allow-list only works if Traefik sees the real client address. The networking stack Ankra generates on UpCloud already trusts PROXY protocol v2 from the load balancer (ports.websecure.proxyProtocol.trustedIPs); on a provider whose load balancer passes X-Forwarded-For instead, the equivalent is forwardedHeaders.trustedIPs.A spoke’s egress address is its NAT gateway. Read it from inside the spoke with ankra cluster exec <any-pod> -n <namespace> -- curl -s https://ifconfig.me, and expect to update the list when a cluster’s network is recreated.
5

Declare the stack

The CRDs deploy first, then Prometheus, then Tempo (which needs the remote-write target up), then Grafana (which needs both data sources). Every parent is a kind and name pair; the shorthand - addon: tempo is silently dropped.
6

Make it the cluster's metrics source

Ankra’s own metrics pages, insights and the AI read from the cluster’s configured Prometheus. Point the hub at itself in the cluster file, and the hub’s dashboards light up on the next sync:
The spokes run Prometheus in agent mode and hold no queryable data, so they cannot point at themselves. Give them the hub instead: add a second Ingress on the hub for metrics-query.example.com with a basicAuth middleware, register that username and password as a metrics credential in Ankra, and set each spoke’s prometheus_metrics to that endpoint with credential_name and verify_ssl: true. Prometheus covers the credential and the endpoints Ankra accepts.

​
Step 5: Infrastructure cluster - central logs with OpenSearch

The OpenSearch operator runs a three-node cluster with Dashboards. Fluent Bit on the hub ships the hub’s own logs; the spokes ship theirs through a logs-ingest endpoint that exposes exactly /_bulk to allow-listed addresses, each spoke with its own user that can write only its own index prefix.
1

The OpenSearch cluster

Pre-creating the admin password in infra-secrets instead of letting the operator generate one is what lets Fluent Bit on the hub read it from the same Secret. The operator defaults securityConfigSecret to an empty name with its own field manager, so this manifest is declared with force: true in the stack, or server-side apply conflicts on it.
2

Retention and the application log index template

Two ISM policies delete indices by age, and one index template keeps the application indices to a single shard with structured log lines under app as a flat_object, so two applications that use the same key with different types never reject each other’s documents.
ISM attaches a policy version to an index when the index is created. Shortening minIndexAge later applies to new indices on its own; existing ones keep the version they were born with until they age out.
3

One user per sending cluster

The security config is a Secret the operator pushes through a securityadmin job whenever it changes. Anything added through the Dashboards UI is wiped by the next run, so roles live in Git. The relevant part gives each spoke a role that can create and bulk-write its own prefix and nothing else:
The users themselves are OpenSearchUser resources whose password comes from the sealed opensearch-log-shipper-production and -development Secrets in infra-secrets; the same values are what the spokes’ shippers are given in Step 8. A 401 on a spoke means those two copies differ.
4

The logs-ingest endpoint

Traefik terminates the public TLS, re-encrypts to OpenSearch and verifies the operator-generated certificate, so no shipper ever needs insecureSkipVerify.
Only exact /_bulk is routed. _search, _cat, _security and the rest of the REST API are not reachable through this host at all; Dashboards on logs.example.com is the only human entrance, and it is gated in Step 6.
5

Fluent Bit for the hub's own logs

6

Declare the stack and the log source

Then make OpenSearch the cluster’s log source so Ankra’s log search and the AI read from it. The credential is registered in the platform and referred to by name; it is never in Git.
The spokes get the same block with their own index pattern once Step 8 has them shipping. Log Sources has the field mappings OpenSearch needs.
If you would rather not run a log store, Ankra Hosted Logs keeps seven days of every cluster’s container logs with one switch. The OpenSearch route is for longer retention, your own retention rules and a store you can query from Grafana and your own tooling.

​
Step 6: Infrastructure cluster - a staff sign-in realm with Kratos and Hydra

Grafana, OpenSearch Dashboards, Harbor and OneUptime each have their own login. Instead of four sets of local accounts, the hub runs one staff identity realm: Ory Kratos, with Microsoft 365 as its only sign-in method, pinned to your Entra tenant; Ory Hydra, which turns a Kratos session into OpenID Connect tokens for each tool; and oauth2-proxy, a forward-auth gate that any Ingress opts into with one annotation. Disabling someone in Entra ends their next sign-in everywhere, because Kratos holds no password, passkey or code of its own that could outlive it.
This realm is for the tools on the hub. Sign-in to Ankra itself is the organisation’s single sign-on, and kubectl access is a cluster access grant. The three are separate on purpose: a person removed from one is not implicitly removed from the others, so remove them from all three.
1

A database for the realm

Kratos and Hydra need separate databases (their table names collide). One CloudNativePG cluster with two instances hosts both; the second instance is a standby for failover. The cloudnative-pg operator can be shared with Harbor’s database in Step 7.
2

Kratos: one sign-in method, pinned to your tenant

Everything Kratos would own outright stays off. Registration is enabled only so a first Microsoft sign-in creates the identity; with the identity mapper supplying every required trait, no form is ever shown.
Three layers decide who gets in: the tenant pin (microsoft_tenant), the schema’s email pattern (which also keeps B2B guests of the tenant out), and each tool’s own domain check. The identity mapper is a small Jsonnet document that reads email, falling back to preferred_username and upn because Entra omits email for many directory members, lower-cases it, and records it as a verified address:
verified_addresses is what makes the consent step emit email_verified=true; without it oauth2-proxy refuses the sign-in. Validate any change to the mapper with jsonnet before merging: Microsoft is the only method, so a mapper error is a total lockout.In Entra, register a single-tenant Web application whose redirect URI is exactly https://staff-login.example.com/self-service/methods/oidc/callback/microsoft, with delegated openid profile email and admin consent. Until that URI is registered Microsoft answers every sign-in with AADSTS50011. Seal the secret: ankra cluster encrypt addon kratos --key client_secret -f <cluster.yaml>.
3

Hydra: tokens for the tools

Two details break everything when wrong. The issuer string, trailing slash included, is the iss claim and oauth2-proxy compares it byte for byte with its --oidc-issuer-url. And allow_termination_from must contain the pod CIDR, or Hydra silently emits http:// redirect and issuer URLs behind Traefik.Each tool is an OAuth2Client resource that hydra-maester registers; its secret is read from a sealed Secret in the tool’s own namespace. Keep skipConsent: false on every client: the consent app is what injects the email claim into the id_token.
4

oauth2-proxy: the gate

The cookie is set for .example.com, so one sign-in covers every gated host for eight hours, then Microsoft again. The gate itself is a Traefik middleware:
Any Ingress on the cluster opts in with one annotation; this is the line already on Grafana’s Ingress in Step 4:
5

Protect the header

Grafana trusts X-Auth-Request-Email as the signed-in user. Only Traefik may present it, so a NetworkPolicy lets only the traefik and prometheus-stack namespaces reach the Grafana pods. Without it, any pod in the cluster could forge the header and become any user.
6

Route the realm and declare the stack

Hydra’s public endpoints (/oauth2, /userinfo, /.well-known/openid-configuration, /.well-known/jwks.json), Kratos’s public endpoints (/self-service, /sessions, /schemas, /.well-known/ory) and the login UI (everything else) share staff-login.example.com as three Ingresses with Traefik router priorities. Neither admin API is routed: Kratos’s admin API has no authentication of its own, and exposing it is publishing the staff directory.
The parent edges matter here more than anywhere: Hydra’s migration job is a Helm pre-install hook, so Ankra must not start the release until CloudNativePG reports the database healthy, and oauth2-proxy must not start until Hydra can answer its discovery request.
Never gate the registry host. A browser redirect on registry.example.com breaks docker login, CI pushes and Ankra’s own chart pulls. Harbor and OpenSearch Dashboards sign in through the realm with their native OpenID Connect support (an OAuth2Client each); only tools with no OIDC of their own go behind the oauth2-proxy gate. Keep every tool’s local admin as the break-glass account, reachable off the gated path.

​
Step 7: Infrastructure cluster - registry and uptime

Two more stacks follow the same pattern, so only what is different is shown.
1

Harbor on CloudNativePG and Redis

Harbor’s chart runs its own database and Redis by default; for production they come from operators instead, so they are backed up, replicated and upgraded like every other database. Harbor’s OIDC settings go in core.extraEnvVars as CONFIG_OVERWRITE_JSON, read from a sealed Secret, with the Hydra client harbor registered by an OAuth2Client next to it:
Harbor applies that JSON on every core start and then makes every setting read-only in the UI, so changes are a Git change plus a bump of a pod annotation. Robot accounts never look at auth_mode, so CI pushes and Ankra’s pulls are unaffected. Once Harbor is up, register it as your application registry so Ankra mints a push robot per application.Add a PrometheusRule beside it. The infrastructure Prometheus evaluates it, so the alert reaches the same channel as everything else:
The first rule is per pod on purpose: a single replica returning 5xx behind a healthy Service is exactly what a readiness probe misses.
2

OneUptime behind the gate

OneUptime probes the public URLs that matter from inside the cluster every minute, opens an incident when one fails, and hosts status pages. Its Ingress carries the sso-gateway-sso@kubernetescrd middleware, and its own login is Global OIDC against the staff Hydra, so staff get in with Microsoft 365 and nobody else sees the page. Its Postgres is another CloudNativePG cluster, two instances, on the retain storage class.A status page meant for customers needs an un-gated host of its own; everything under the gated host is staff-only by construction.

​
Step 8: Production and development clusters - send metrics and logs

The spokes carry no Grafana, no Alertmanager, no rules and no queryable Prometheus. Two small stacks make each of them a sender. Add the following to clusters/production-<short-id>/production.yaml, then repeat for development with the names swapped.
1

Prometheus in agent mode

The same kube-prometheus-stack chart, with agentMode on and the hub as the remote-write target. The cluster external label is what tells this cluster’s series apart in the central store; keep it equal to the Ankra cluster name.
Every ServiceMonitor and PodMonitor an application ships is picked up (the NilUsesHelmValues: false lines), so adding an exporter to an application is enough for it to appear in Grafana on the hub.
2

Fluent Bit shipping to the hub

Read only the application namespaces, buffer on the node’s filesystem so an outage on the hub is replayed rather than lost, and authenticate as this cluster’s own OpenSearch user. The password is the same value sealed into the hub’s infra-secrets stack in Step 3.
Indices arrive on the hub as production-YYYY.MM.DD, one per cluster per day, filtered by kubernetes.namespace_name or cluster. Structured JSON lines land under app.*; anything else stays in log.
3

Add the spoke's egress address on the hub

Both ingest endpoints refuse everything that is not on their allow-list. Read the spoke’s egress address, add it to metrics-ingest-allowlist and logs-ingest-allowlist in the hub’s cluster file, and apply. Until then every write from the spoke is a 403 in the agent’s and Fluent Bit’s logs.
4

Application databases on the spokes

Every stateful application stack on a spoke follows one shape: a CloudNativePG cluster with three instances spread across workers, the retain storage class, and an owner role per database. This is what makes Step 9 work, because the backup data plane captures CloudNativePG clusters by name.

​
Step 9: Back up every stateful stack

A vault is where restore points land; a stack’s backup policy is what puts them there. Protection is a property of the stack itself: the command writes a backup block onto the stack definition, and the platform installs the backup data plane (Velero and the CloudNativePG Barman plugin) on the cluster the first time a stack on it is protected.
1

See what a backup would carry

The output lists the stack’s standalone PersistentVolumeClaims and the databases an operator runs, with the engine that would capture each and the consistency it can promise. Databases are captured by default; volumes only where you name them.
2

Protect each stateful stack, staggered

One stack at a time, each on its own minute, so the captures do not compete for the same disks and the same vault at once:
The first command reports Backup stack: installing until Velero is actually on the cluster; a capture started before then waits rather than failing. The policy lands in the cluster file as a backup block, so from here on it is reviewed and versioned like everything else:
Repeat on the development cluster against development-backups. The hub’s own stateful stacks - sso-gateway, registry, uptime - are protected the same way into a third vault, or into production-backups if you prefer fewer buckets.
3

Read the posture

Every stack on the cluster is listed against what protects it: protected, unprotected (no_data_assets) for stacks with nothing to carry, and unprotected (no_policy) for the ones you still owe a policy. The last line of each row says when the last run succeeded or failed and when the next fires. A stack that holds data and reads no_policy is the thing to fix before calling the estate production.
4

Prove a restore point can be read

get shows the manifest, the assets it carries and the run that produced it. A restore point whose status is failed carries nothing; find the run under ankra runs list --kind backup --cluster production and fix the cause before the next schedule fires. Backups and Restore Points covers restoring in place and cloning a stack with its data onto another cluster.
Nothing is scheduled inside the cluster: no Velero Schedule, no CloudNativePG ScheduledBackup. Ankra takes the scheduled restore points itself, so every backup is one you can list, inspect and restore from the platform. If you add an in-cluster schedule as well, the same data is captured twice.

​
Step 10: Security and access

Most of this is already in place from Step 1. What remains is turning it on where it is off by default, and deciding who reaches what.
1

Security baseline on every cluster

Clusters Ankra provisions arrive with the ankra-security-baseline stack: Trivy Operator for image vulnerabilities, configuration audits and the CIS benchmark, and Pod Security Standards enforced by the API server from namespace labels the agent manages. It starts in Audit mode. Once the Security Center shows no workloads failing the baseline standard, enforce it:
Kyverno is not part of the baseline; it is the opt-in ankra-policy-engine profile for image-signature verification and policies of your own. Cluster Security describes what runs where.
2

Kubernetes access through the gateway, time-boxed

Nobody gets a static kubeconfig. Access to a cluster’s API goes through the Ankra kube gateway, per person, per role, with an expiry and a reason:
The organisation access policy can require an expiry and a reason on every grant. For an incident at three in the morning, ankra cluster access elevate --cluster production --role edit --expires 4h --reason "incident 4711" gives the caller themselves time-boxed access within that policy, and the grant is in the audit log.
3

Credentials that can do one thing

  • The DNS credential can edit one zone.
  • The Git credential can write one repository.
  • Each log shipper user can create and bulk-write one index prefix.
  • The metrics-ingest endpoint can append samples and nothing else.
  • Harbor robot accounts are minted per application by Ankra and can push one project.
  • The backup vault keys are held by Ankra and never printed.
None of them is reused, so revoking one revokes one thing.
4

Everything sensitive is sealed before it is committed

Run ankra cluster sops-config once to confirm the organisation has its AGE key, then make a rule of it: no stringData in a commit, no webhook URL in values. The pre-push check is the simplest one there is:

​
Verify

Run through this once at the end, and again after any change to the hub.
1

Every stack converged

Every stack reads up; sso-postgres x2, hydra, kratos, the UI and oauth2-proxy are Running; three OpenSearch nodes and two Dashboards are Ready.
2

The spokes' series have arrived

Three values, one per cluster. A missing cluster is an allow-list or a cluster label problem; the spoke’s Prometheus log shows which.
3

The ingest endpoints refuse everyone else

From your laptop, which is not on the allow-list:
403, 403, 404. The third line is the path restriction: the REST API is not reachable through that host at all.
4

The realm answers

Exactly https://staff-login.example.com/, trailing slash included. Then in a browser, from https://metrics.example.com/: the Microsoft 365 sign-in, and Grafana with your address in the top right. Start a sign-in from a tool, never from staff-login itself.
5

Logs are searchable from Ankra

Open the production cluster’s Logs and search cluster:production. Rows appear with kubernetes.namespace_name and app.* fields. In OpenSearch Dashboards the same query runs against production-*.
6

Backups are on and recent

No stack with data reads no_policy, every last run is succeeded, and both vaults read ready.
7

An alert reaches the channel

The test lands in the channel; the preview names the destination. Then silence the Alertmanager Watchdog route for a minute in the hub’s values and watch Alertmanager’s own Watchdog arrive the same way - that proves the in-cluster path end to end.

​
Troubleshooting

The spoke’s egress address is not on the hub’s allow-list. The address changes when a cluster’s network is recreated. Read it from inside the spoke, update metrics-ingest-allowlist and logs-ingest-allowlist in the hub’s cluster file, and apply. Fluent Bit replays its filesystem buffer once the 403s stop; Prometheus’s remote-write queue holds roughly an hour before it drops samples.
The password in the spoke’s central-logs-opensearch Secret and the hub’s opensearch-log-shipper-<cluster> Secret differ. Decrypt both with ankra cluster decrypt manifest <name> -f <cluster.yaml> and compare. The shipper users are also briefly absent for up to 30 seconds after every security config push while the operator re-creates them; a burst of 401s right after a commit to opensearch-securityconfig is that, and clears on its own.
The shorthand - addon: name parses to an empty parent and is dropped without an error. Every parent is kind and name on two lines. Confirm what was stored with ankra cluster stacks list <stack> -o json, which echoes each resource’s real parents.
Dashboards fetches the realm’s discovery document at startup and refuses to start while staff-login.example.com is unreachable, password form included. Fix the realm first. The REST API with the admin basic credentials still works in the meantime through ankra cluster exec on an OpenSearch pod.
Two separate causes with the same symptom. allow_termination_from must contain the pod CIDR, or Hydra does not trust X-Forwarded-Proto from Traefik and emits http:// everywhere. And the issuer string in Hydra’s values and oauth2-proxy’s --oidc-issuer-url must be identical, trailing slash included.
The redirect URI https://staff-login.example.com/self-service/methods/oidc/callback/microsoft is not registered on the Entra application, platform Web. Keep pkce: never in the Kratos provider: with PKCE on, Kratos switches to the provider-less callback path and the registered URI no longer matches.
Alertmanager’s receiver still holds a placeholder, or the sealed webhook_url was never declared in encrypted_paths and deployed as ciphertext. ankra cluster addons values kube-prometheus-stack --cluster infrastructure shows the live values; the webhook must be a real URL there. Ankra’s own notifications are a separate path: ankra alerts destinations test proves that one.
ankra runs list --kind backup --cluster production names the run; ankra runs get <run-id> shows which asset failed. The usual causes are a VolumeSnapshotClass whose CSI sidecars cannot write snapshot status (a chart RBAC gap - upgrade the CSI add-on), and a database that is backup-policy-annotated but whose operator plugin is not yet installed (the data plane shows installing in ankra cluster backups status).
A manifest was renamed, or a namespace manifest was removed, with a StorageClass that reclaims on delete. The retain class in Step 3 is the guard; the PersistentVolume still exists and can be re-bound. Protect Persistent Data walks through it.

​
Next steps

Turn the hub into a stack profile

Parameterise the domain, the tenant and the storage class, and launch the same infrastructure for the next region in one apply.

Ship an application to production

Connect a repository, build and scan on Ankra Pipelines, publish to the Harbor you just built, and deploy to the production cluster on merge.

Connect the clusters with Cluster Mesh

Let a service on production reach the hub’s Tempo or OpenSearch over the WireGuard overlay instead of a public endpoint.

Compliance evidence

Map the scanner coverage, the access grants and the backup posture you now have onto a framework.