Prometheus

Connect your Prometheus instance to Ankra to view metrics directly in the platform. This powers resource graphs, the AI Assistant’s metrics awareness, and cluster health indicators.

Why Connect Prometheus?

When you connect Prometheus to Ankra:

Resource Metrics - View CPU, memory, and network graphs on pods, deployments, and nodes
AI-Powered Analysis - The AI Assistant can query your metrics to diagnose issues
Cluster Overview - See aggregate metrics on the cluster dashboard
Troubleshooting - Correlate metrics with logs and events in one place

Supported Endpoints

Ankra connects to any Prometheus-compatible endpoint:

Provider	Example Endpoint
Self-hosted Prometheus	`http://prometheus.monitoring.svc.cluster.local:9090`
Prometheus via port-forward	`http://localhost:9090`
Grafana Cloud	`https://prometheus-prod-xx-xxx.grafana.net`
Amazon Managed Prometheus	`https://aps-workspaces.<region>.amazonaws.com/workspaces/<id>`
Google Cloud Managed Prometheus	`https://monitoring.googleapis.com/v1/projects/<project>`
Thanos Query	`http://thanos-query.monitoring.svc.cluster.local:9090`
Victoria Metrics	`http://victoria-metrics.monitoring.svc.cluster.local:8428`

Connecting Prometheus

Open Data Sources

Navigate to your cluster → Settings → Data Sources.

Select Metrics

Choose Metrics in the sidebar.

Pick a provider

Under Provider, select Prometheus for a self-hosted instance, or Grafana Cloud / Mimir or VictoriaMetrics for a hosted Prometheus-compatible endpoint. The managed cloud providers (AWS, Google, Azure) are marked Coming soon.

Enter the endpoint

Click Connect Existing and enter your Prometheus endpoint URL. For a self-hosted Prometheus, Ankra also scans the cluster for Prometheus services and offers the ones it finds.For in-cluster Prometheus, use the internal service URL:

http://prometheus-server.monitoring.svc.cluster.local:9090

For kube-prometheus-stack:

http://prometheus-operated.monitoring.svc.cluster.local:9090

Add credentials (if required)

If your Prometheus requires authentication, expand Authentication, enter a Credential Name, and pick the Authentication Method:

API Token - For bearer token authentication
User Account - For basic auth (username/password)

Test and save

Click Test Connection to verify Ankra can reach your Prometheus instance, then Submit to store the configuration.

Authentication Options

No Authentication
API Token
Basic Auth

For Prometheus instances without authentication (common for in-cluster deployments): Simply enter the endpoint URL and save. No credentials needed.

For Prometheus instances using bearer token authentication:

Expand Authentication in the metrics form and enter a Credential Name
Choose the API Token method
Enter your token value

Ankra sends the token as: Authorization: Bearer <token>

In-Cluster vs External Prometheus

In-Cluster Prometheus

If Prometheus runs inside the same cluster:

Use the internal Kubernetes service URL
No authentication is typically needed
The Ankra agent forwards requests through the cluster network

Common service URLs:

# Standard Prometheus
http://prometheus-server.monitoring.svc.cluster.local:9090

# kube-prometheus-stack
http://prometheus-operated.monitoring.svc.cluster.local:9090

# Prometheus Operator
http://prometheus-k8s.monitoring.svc.cluster.local:9090

External Prometheus

If Prometheus runs outside the cluster (cloud managed, separate infrastructure):

Use the external URL (must be reachable from your cluster)
Configure authentication as required by your provider
Ensure network connectivity and firewall rules allow access

Credentials

In the portal, the credential is created with the metrics source, in the Authentication section of the form:

Credential Name - A descriptive name like prometheus-prod
Authentication Method - API Token for bearer token auth, User Account for basic auth

Ankra keeps its own copy of the secret and sends it with every query and with the health probe that runs every five minutes. It never reads the credential from the cluster - not from a Kubernetes Secret, not from the Prometheus configuration - so rotating the password on the Prometheus side does not update Ankra, and updating Ankra does not change Prometheus.

A saved credential cannot be edited in place. To rotate a token or password, delete the credential from Settings → Data Sources → Metrics and create a new one.To keep the gap short, rotate Prometheus first, then Ankra: once the new secret is live on the Prometheus side, open Edit on the metrics source, type the new credentials into the Authentication section and click Test Connection. The test uses what you typed without saving it, so a green result proves the new secret opens the door before you replace the saved credential. Queries fail with a 401 for the time between Prometheus dropping the old secret and Ankra receiving the new one.

Rotating by API

The portal path needs a browser. A scripted rotation re-submits the cluster’s import request with the new secret. POST /api/v1/clusters/import is create-or-update and merges by member: on a cluster that already exists it applies each spec member the body carries and leaves the members it omits exactly as they are. A body holding only prometheus_credential touches nothing else - not the stacks, add-ons and manifests, not the GitOps binding, not the metrics source and its endpoint. The one thing that runs the other way is a stack that is in the body: it replaces the saved stack of that name whole, so leave stacks out of a credential-only call. A spec.prometheus_credential block replaces the cluster’s saved metrics credential in place: same credential, new secret. The credential itself never reaches Git.

curl -sS -X POST "https://platform.ankra.app/api/v1/clusters/import?wait=true" \
  -H "Authorization: Bearer $ANKRA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "<cluster-name>",
    "spec": {
      "prometheus_credential": {
        "name": "<credential-name>",
        "username": "<username>",
        "password": "<new-password>"
      }
    }
  }'

The token is a REST token - a personal access token created without --scopes, or a service token with no MCP scope - belonging to a role that can change the cluster. A token carrying mcp:read or mcp:write reaches only the MCP server and is refused here with 403.
Send the whole block. It replaces the saved credential rather than patching it: name, username and password for basic auth, or name and token for bearer auth. An omitted username is saved as empty and basic auth stops working.
Keep the name the Metrics panel shows, so the stored secret is overwritten rather than left behind under the old name. A cluster holds one metrics credential, and a different name still replaces that one.
Use wait=true. Without it the call answers 202 at once and whatever goes wrong afterwards is visible only in platform logs. With it the call blocks until the write is done and answers 200 with the outcome (check its errors list) or a 4xx naming the rejection.
Expect one Git commit after a hand rotation. On a cluster connected to a Git repository the call ends with the platform’s routine push of the cluster state. A plain file is written only when it differs from the branch, and a SOPS-encrypted file only when the # ankra_content_fingerprint header Ankra stamped on it no longer matches the values Ankra holds, so on a cluster whose files Ankra last wrote itself the push commits nothing. A rotation breaks that steady state on purpose: the manifest holding Prometheus’s own basic-auth Secret was just re-sealed by your sops run, Ankra imported the rotated values from your commit, and the header on the file - kept from the old seal or dropped - no longer matches them. The push re-encrypts that manifest (same values, new ciphertext: every sops run draws a fresh data key) and rewrites .ankra-owned.json beside it. The commit is titled Update cluster configuration, authored by ankra-platform[bot], and touches nothing else. To check that yourself, fetch the cluster’s IaC export with the same token - GET /api/v1/org/clusters/imported/{cluster_id}/iac, decode yaml_string_base64 - before and after the call: the two documents are identical.
Let Ankra sync your rotation commit first. The push refuses to touch a file that changed in Git since the last commit Ankra synced (Monitoring Sync Status shows which one that is). A call that lands before the sync has caught up therefore answers 422 Failed to push changes to Git repository - after the credential has been saved. Do not retry: verify with the query below; the reconciler imports your commit on its next pass and Ankra’s next write re-seals the manifest then. To skip the push altogether, send your existing prometheus_metrics block verbatim alongside the credential: a save that changes nothing outside the metrics members never pushes, and the re-seal waits for Ankra’s next write instead.

ankra cluster apply drops prometheus_credential from the YAML before sending it, and the Git sync reads only git_repository, prometheus_metrics, log_source and stacks from spec, so a prometheus_credential block in the cluster file is ignored. Neither creates a second credential - a cluster holds one - and neither updates it: this call is the only scripted path to the secret. To make Git name the credential in use, set prometheus_metrics.credential_name to the saved credential’s name; the sync refuses a name that resolves to nothing. There is no API twin of Test Connection: the portal is the one place a credential can be tried without saving it. After the call, ankra cluster metrics query 'up' against the cluster proves the whole path - the saved credential, the agent, Prometheus - in one read. Query results are cached for ten seconds, so vary the query or wait before reading the new state.

Managing the Connection in Git

If the cluster is connected to a GitOps repository, the metrics data source is part of the cluster document at clusters/<cluster-name>-<cluster-id>/<cluster-name>.yaml. Adding or editing the spec.prometheus_metrics block connects and updates the source on the next sync, exactly as the settings page does:

spec:
  prometheus_metrics:
    endpoint: http://prometheus-operated.monitoring.svc.cluster.local:9090
    flavor: self_hosted
    credential_name: prometheus-prod

Accepted fields are endpoint, credential_name, flavor and verify_ssl - see the ImportCluster schema for what each one means. Any other top-level field is refused with an error naming it, rather than ignored. A block without credential_name keeps using the credential already saved on the cluster, if there is one; the Metrics panel under Settings → Data Sources shows which.

Credentials never go in Git. A username, password, token or auth_scheme inside the block is refused by name: silently ignoring it would leave a live secret in your repository while you believe it is in effect. Create the credential in the platform and reference it with credential_name.Removing the block does not disconnect Prometheus. Git can create and update the source, never disconnect it - the source survives the sync and the block reappears the next time Ankra regenerates the file. Disconnect from Settings → Data Sources instead.

Verifying the Connection

After connecting Prometheus:

Check Data Sources - The Metrics entry in the sidebar shows a green dot, and the panel shows the last health check
View Resource Metrics - Go to any pod or deployment and check for metric graphs
Ask the AI - Press ⌘+J and ask “What’s the CPU usage of this pod?”

If metrics aren’t showing:

Verify the endpoint URL is correct
Check that Prometheus has data for your cluster
Re-test the connection from Settings → Data Sources → Metrics → Edit
Check network connectivity between your cluster and Prometheus

Troubleshooting

Connection Test Fails

Verify the URL is correct and includes the port
For in-cluster Prometheus, ensure the service exists:
kubectl get svc -n monitoring

Check if Prometheus is running:

kubectl get pods -n monitoring -l app=prometheus

Test connectivity from within the cluster:

kubectl run curl --rm -it --image=curlimages/curl -- curl http://prometheus-server.monitoring:9090/api/v1/query?query=up

Authentication Errors

Verify the credential is correctly configured
For API tokens, ensure the token hasn’t expired
For basic auth, verify the username and password are correct
Check Prometheus logs for authentication failures

Metrics Not Appearing

Prometheus may not have metrics for the resource yet (wait for scrape interval)
Verify Prometheus is scraping the correct targets
Check the Prometheus UI targets page for scrape errors
Ensure the metric names match what Ankra expects (standard Kubernetes metrics)

Slow Queries

Large time ranges can be slow. Try narrower ranges.
Consider using recording rules for frequently-queried metrics
Check Prometheus resource usage (may need more CPU/memory)

What Metrics Does Ankra Use?

Ankra queries standard Kubernetes and node metrics:

Metric	Source	Used For
`container_cpu_usage_seconds_total`	cAdvisor	Pod/container CPU graphs
`container_memory_working_set_bytes`	cAdvisor	Pod/container memory graphs
`node_cpu_seconds_total`	node-exporter	Node CPU usage
`node_memory_MemAvailable_bytes`	node-exporter	Node memory usage
`kube_pod_status_phase`	kube-state-metrics	Pod status indicators
`kube_deployment_status_replicas`	kube-state-metrics	Deployment health

If you’re using kube-prometheus-stack, all these metrics are included by default.

Next Steps

Build a Monitoring Stack

Deploy Prometheus and Grafana with one Stack.

AI Assistant

Use the AI to query and analyze your metrics.

Kubernetes Insights

Explore built-in cluster dashboards.

Credentials

Manage authentication for all integrations.

Get Started

Concepts

Provision & Import

Operate

Security & Compliance

Deliver

Platform

AI

Kubernetes Resources

Reference

Integrations

Why Connect Prometheus?

Supported Endpoints

Connecting Prometheus

Authentication Options

In-Cluster vs External Prometheus

In-Cluster Prometheus

External Prometheus

Credentials

Rotating by API

Managing the Connection in Git

Verifying the Connection

Troubleshooting

What Metrics Does Ankra Use?

Next Steps

Build a Monitoring Stack

AI Assistant

Kubernetes Insights

Credentials

​Why Connect Prometheus?

​Supported Endpoints

​Connecting Prometheus

​Authentication Options

​In-Cluster vs External Prometheus

​In-Cluster Prometheus

​External Prometheus

​Credentials

​Rotating by API

​Managing the Connection in Git

​Verifying the Connection

​Troubleshooting

​What Metrics Does Ankra Use?

​Next Steps

Build a Monitoring Stack

AI Assistant

Kubernetes Insights

Credentials

Why Connect Prometheus?

Supported Endpoints

Connecting Prometheus

Authentication Options

In-Cluster vs External Prometheus

In-Cluster Prometheus

External Prometheus

Credentials

Rotating by API

Managing the Connection in Git

Verifying the Connection

Troubleshooting

What Metrics Does Ankra Use?

Next Steps