> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ankra.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Alert on Manifest Failures

> Set up Slack notifications when manifests fail to deploy or don't reach a healthy state

<Note>
  This guide walks you through setting up alerts for manifest deployment failures with Slack notifications. You'll create a Slack webhook and configure an alert rule to monitor specific clusters.
</Note>

## What You'll Build

By the end of this guide, you'll have:

* A Slack webhook integration connected to Ankra
* An alert rule that triggers when manifest jobs fail
* Notifications sent to your Slack channel when deployments fail
* Automatic AI-powered root cause analysis for every failure

<Frame>
  ```mermaid theme={null}
  flowchart LR
      A[Manifest Deploy] --> B{Job Status}
      B -->|Failed| C[Alert Triggered]
      C --> D[Slack Notification]
      C --> F[AI Analysis Job]
      F --> G[AI Incident Created]
      B -->|Success| E[No Alert]
  ```
</Frame>

***

## Prerequisites

Before you begin, ensure you have:

* An Ankra account with access to your organisation's alerts
* Admin access to a Slack workspace (to create an app)
* At least one cluster with manifests deployed

***

## Part 1: Create a Slack Webhook

### Step 1: Create a Slack App

1. Go to [api.slack.com/apps](https://api.slack.com/apps)
2. Click **Create New App**
3. Choose **From scratch**
4. Enter the app details:
   * **App Name:** `Ankra Alerts`
   * **Workspace:** Select your workspace
5. Click **Create App**

### Step 2: Enable Incoming Webhooks

1. In your app settings, click **Incoming Webhooks** in the left sidebar
2. Toggle **Activate Incoming Webhooks** to **On**
3. Click **Add New Webhook to Workspace**
4. Select the channel for alerts (e.g., `#deployments` or `#alerts`)
5. Click **Allow**

### Step 3: Copy the Webhook URL

After authorization, copy your webhook URL. It will look like:

```
https://hooks.slack.com/services/TXXXXX/BXXXXX/your-webhook-token
```

<Warning>
  Keep this URL secret. Anyone with it can post to your channel.
</Warning>

### Step 4: Create the Webhook in Ankra

1. In Ankra, navigate to **Alerts** in the sidebar
2. Click the **Integrations** tab
3. Click **Create Webhook**
4. Fill in the details:

| Field           | Value                               |
| --------------- | ----------------------------------- |
| **Name**        | `Slack Deployment Alerts`           |
| **URL**         | Paste your Slack webhook URL        |
| **Description** | Notifications for manifest failures |
| **Template**    | Select **Slack**                    |

5. Click **Create**

### Step 5: Test the Webhook

1. On the webhook detail page, click **Send Test**
2. Check your Slack channel for the test message
3. If it appears, your webhook is ready!

<Tip>
  The default Slack template includes alert name, severity, resource info, and a link to Ankra. You can customize it later if needed.
</Tip>

***

## Part 2: Create the Alert Rule

Now create an alert rule that monitors manifest deployments on specific clusters.

### Step 1: Start Creating an Alert

1. Go to **Alerts** in Ankra
2. Click the **Alert Rules** tab
3. Click **Create Alert**

### Step 2: Configure Basic Details

In the **Details** step:

| Field           | Value                                                |
| --------------- | ---------------------------------------------------- |
| **Name**        | `Manifest Deployment Failed`                         |
| **Description** | Alert when manifest jobs fail on production clusters |
| **Enabled**     | Toggle **On**                                        |

Click **Next** to continue.

### Step 3: Select Resource Type

In the **Resource** step:

1. **Resource Type:** Select **Cluster Resource**
2. **Cluster Resource Kind:** Select **Manifest**

This tells Ankra to monitor manifest resources specifically.

### Step 4: Select Specific Clusters

Under **Clusters**, select the clusters you want to monitor:

1. Click the cluster dropdown
2. Check the boxes for your target clusters (e.g., `production-cluster`, `staging-cluster`)
3. Only selected clusters will trigger this alert

<Note>
  Leave all clusters unchecked to monitor manifests across your entire organisation.
</Note>

### Step 5: Configure the Condition

In the **Conditions** section, set up the failure condition:

**Condition 1:**

| Field              | Value      |
| ------------------ | ---------- |
| **Condition Type** | Job Status |
| **Operator**       | Equals     |
| **Value**          | Failed     |

This condition triggers when any manifest job enters a `Failed` state.

<Accordion title="Optional: Add additional conditions">
  You can add multiple conditions with AND/OR logic:

  **Example: Alert on both failures and timeouts**

  * Condition 1: Job Status = Failed
  * **OR**
  * Condition 2: Job Status = Timeout

  **Example: Alert only when stuck for a while**

  * Condition 1: Job Status = Failed
  * **AND**
  * Condition 2: Stuck in State > 5 minutes
</Accordion>

Click **Next** to continue.

### Step 6: Set Severity and Cooldown

In the **Severity** step:

| Field        | Value        | Description                                |
| ------------ | ------------ | ------------------------------------------ |
| **Severity** | `critical`   | High priority for deployment failures      |
| **Cooldown** | `15 minutes` | Prevent repeated alerts for the same issue |

Click **Next** to continue.

### Step 7: Attach the Slack Webhook

In the **Notifications** step:

1. Under **Webhooks**, find your `Slack Deployment Alerts` webhook
2. Check the box to enable it
3. The alert will now send notifications to Slack when triggered

Click **Create** to save the alert rule.

***

## What Happens When a Manifest Fails

When a manifest deployment fails on one of your selected clusters:

<Frame>
  ```mermaid theme={null}
  sequenceDiagram
      participant M as Manifest
      participant A as Alert Engine
      participant W as Webhook
      participant AI as AI Analysis

      M->>A: Job Status = Failed
      A->>A: Evaluate conditions
      A->>W: Send notification
      W->>W: POST to Slack
      A->>AI: Create AI Analysis Resource
      AI->>AI: Schedule Analysis Job
      AI->>AI: Collect pods, events, logs
      AI->>AI: Generate root cause analysis
      AI->>A: AI Incident ready
  ```
</Frame>

### Step-by-Step Breakdown

1. **Ankra detects the failure** - Job status changes to `Failed`
2. **Alert rule evaluates** - Condition matches (Job Status = Failed)
3. **Cooldown check** - If no recent alert for this resource, proceed
4. **Webhook fires** - Slack receives the notification immediately
5. **AI Analysis Resource created** - Ankra creates a tracking resource for the analysis
6. **AI Analysis Job scheduled** - A background job starts collecting data
7. **Data collection** - The job gathers:
   * Pod status and container states
   * Kubernetes events (warnings, errors)
   * Container logs (last 50 lines)
   * Job results and error messages
8. **AI Incident generated** - Ankra's AI analyzes the data and produces:
   * Root cause identification
   * Severity assessment
   * Recommended actions
   * Links to affected resources

<Note>
  **View AI Incidents:** Navigate to **Alerts → AI Incidents** tab to see all generated analyses. Each incident includes the AI's findings, affected resources, and suggested next steps. You can also ask follow-up questions directly in the incident view.
</Note>

### Example Slack Notification

Your Slack channel will receive a message like:

```
🔴 Alert: Manifest Deployment Failed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity: critical
Resource: nginx-config (Manifest)
Cluster: production-cluster

Triggered at: Dec 15, 2024, 2:34 PM

🔗 View in Ankra
```

***

## Verifying Your Setup

Test your configuration end-to-end:

1. **Create a test manifest** with an intentional error (e.g., invalid YAML)
2. **Deploy the manifest** to one of your monitored clusters
3. **Watch for the failure** in the Operations page
4. **Check Slack** for the alert notification
5. **Review the AI Incident** for root cause analysis

***

## Additional Alert Scenarios

<Accordion title="Alert on manifest state changes">
  To alert when manifests enter an unhealthy state (not just job failures):

  | Field              | Value          |
  | ------------------ | -------------- |
  | **Condition Type** | Resource State |
  | **Operator**       | Equals         |
  | **Value**          | Down           |

  This triggers when a manifest's health status becomes "Down".
</Accordion>

<Accordion title="Alert on stuck deployments">
  To alert when manifests are stuck in a deploying state:

  | Field              | Value                     |
  | ------------------ | ------------------------- |
  | **Condition Type** | Stuck in State (duration) |
  | **Operator**       | Greater than              |
  | **Value**          | 10 (minutes)              |

  This triggers when a manifest has been in a transitional state for too long.
</Accordion>

<Accordion title="Alert on repeated failures">
  To alert only after multiple failures:

  | Field              | Value                 |
  | ------------------ | --------------------- |
  | **Condition Type** | Failed Job Count      |
  | **Operator**       | Greater than or equal |
  | **Value**          | 3                     |

  This triggers after 3 or more failed jobs on the same manifest.
</Accordion>

***

## Troubleshooting

### Alert not triggering

* Verify the alert rule is **Enabled**
* Check that the correct **clusters** are selected
* Ensure the manifest is in a cluster you're monitoring
* Confirm the **condition** matches (Job Status = Failed)

### Slack message not appearing

* Test the webhook using the **Send Test** button
* Verify the webhook URL is correct
* Check that the Slack app still has channel access
* Look for errors in the webhook's delivery history

### Too many alerts

* Increase the **cooldown period** (e.g., 30 minutes)
* Add more specific conditions to filter alerts
* Consider monitoring only production clusters

***

## Related

* [Alerts](/guides/alerts) - Full alert rule configuration reference
* [Webhooks](/guides/webhooks) - Template variables and custom payloads
* [AI Incidents](/platform/ai-incidents) - Automatic root cause analysis
* [Operations](/platform/operations) - View job history and logs

***

Still have questions? [Join our Slack community](https://join.slack.com/t/ankra-community/shared_invite/zt-3a5rem8f8-cUho4epX2MoLT83bFf~VSA) and we'll help out.