> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devhelm.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerting Guide

> Set up alert channels, notification policies, and escalation chains to route incidents to your team

This guide walks you through configuring the alerting system end to end. For conceptual background, see [Alerting overview](/alerting/overview).

## Step 1: Create alert channels

Alert channels are the destinations where notifications get sent. Create them first, then reference them in notification policies.

<CodeGroup>
  ```bash CLI theme={null}
  # Slack
  devhelm alert-channels create \
    --name "Engineering Slack" \
    --type slack \
    --webhook-url "https://hooks.slack.com/services/T.../B.../xxx"

  # PagerDuty
  devhelm alert-channels create \
    --name "On-Call PagerDuty" \
    --type pagerduty \
    --config '{"channelType":"pagerduty","routingKey":"your-events-api-v2-routing-key"}'

  # Email
  devhelm alert-channels create \
    --name "Team Email" \
    --type email \
    --config '{"channelType":"email","recipients":["oncall@example.com","team@example.com"]}'
  ```

  ```yaml devhelm.yml theme={null}
  alertChannels:
    - name: Engineering Slack
      config:
        channelType: slack
        webhookUrl: ${SLACK_WEBHOOK_URL}

    - name: On-Call PagerDuty
      config:
        channelType: pagerduty
        routingKey: ${PAGERDUTY_ROUTING_KEY}

    - name: Team Email
      config:
        channelType: email
        recipients:
          - oncall@example.com
          - team@example.com
  ```
</CodeGroup>

### Test a channel

Verify your channel is configured correctly:

```bash theme={null}
devhelm alert-channels test <channel-id>
```

This sends a test notification through the channel and reports success or failure.

## Step 2: Create notification policies

Notification policies connect incidents to alert channels through match rules and escalation chains.

### Simple policy (notify Slack for all incidents)

```bash theme={null}
curl -X POST https://api.devhelm.io/api/v1/notification-policies \
  -H "Authorization: Bearer $DEVHELM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "All incidents to Slack",
    "matchRules": [],
    "escalation": {
      "steps": [
        {
          "delayMinutes": 0,
          "channelIds": ["<slack-channel-id>"]
        }
      ]
    },
    "priority": 0
  }'
```

An empty `matchRules` array is a **catch-all** — it matches every incident.

### Tiered escalation

Route critical incidents through an escalation chain:

```json theme={null}
{
  "name": "Critical escalation",
  "matchRules": [
    { "type": "severity_gte", "value": "DOWN" }
  ],
  "escalation": {
    "steps": [
      {
        "delayMinutes": 0,
        "channelIds": ["<slack-channel-id>"],
        "requireAck": false
      },
      {
        "delayMinutes": 15,
        "channelIds": ["<pagerduty-channel-id>"],
        "requireAck": true,
        "repeatIntervalSeconds": 300
      },
      {
        "delayMinutes": 30,
        "channelIds": ["<email-channel-id>"]
      }
    ],
    "onResolve": "notify_all_steps",
    "onReopen": "restart_from_beginning"
  },
  "priority": 10
}
```

This policy:

1. Immediately notifies Slack
2. After 15 minutes (if not acknowledged), pages PagerDuty and repeats every 5 minutes
3. After 30 minutes, emails the team

### Scoped policies

Use match rules to target specific monitors or conditions:

```json theme={null}
{
  "name": "Production HTTP monitors only",
  "matchRules": [
    { "type": "monitor_type_in", "values": ["HTTP"] },
    { "type": "monitor_tag_in", "values": ["production"] }
  ],
  "escalation": {
    "steps": [
      { "delayMinutes": 0, "channelIds": ["<channel-id>"] }
    ]
  },
  "priority": 5
}
```

### Available match rules

| Rule                   | Matches on                  | Fields                                     |
| ---------------------- | --------------------------- | ------------------------------------------ |
| `severity_gte`         | Severity ≥ value            | `value`: `DOWN`, `DEGRADED`, `MAINTENANCE` |
| `monitor_id_in`        | Specific monitors           | `monitorIds`: list of UUIDs                |
| `monitor_type_in`      | Monitor types               | `values`: `HTTP`, `TCP`, `DNS`, etc.       |
| `monitor_tag_in`       | Monitor tags                | `values`: tag names                        |
| `region_in`            | Affected regions            | `regions`: region codes                    |
| `incident_status`      | Event type                  | `value`: `created`, `resolved`, `reopened` |
| `service_id_in`        | Status data services        | `values`: service IDs                      |
| `resource_group_id_in` | Resource groups             | `values`: group IDs                        |
| `component_name_in`    | Status data component names | `values`: component names                  |

## Step 3: Set policy priority

Policies are evaluated by priority (highest first). **All matching policies execute** — there's no "first match wins". Set higher priority on more specific policies:

| Priority | Policy                   | Match rules                                            |
| -------- | ------------------------ | ------------------------------------------------------ |
| 10       | Critical escalation      | `severity_gte: DOWN`                                   |
| 5        | Production HTTP to Slack | `monitor_type_in: HTTP` + `monitor_tag_in: production` |
| 0        | Everything to email      | *(catch-all)*                                          |

## Testing your setup

1. **Test individual channels**: `devhelm alert-channels test <id>`
2. **Create a test monitor** with a low frequency that will fail:

```bash theme={null}
devhelm monitors create \
  --name "Alert Test" \
  --type HTTP \
  --url https://httpstat.us/500 \
  --frequency 300 \
  --regions us-east
```

3. Wait for the incident to trigger and verify notifications arrive
4. Delete the test monitor when done

## Next steps

<CardGroup cols={2}>
  <Card title="Integration setup" icon="plug" href="/integrations/slack">
    Detailed setup guides for each alert channel type.
  </Card>

  <Card title="Incidents guide" icon="triangle-exclamation" href="/guides/incidents">
    Manage incidents and maintenance windows.
  </Card>
</CardGroup>