> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devhelm.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Tiered Escalation

> Build Slack to PagerDuty to email escalation chains in DevHelm

By the end of this guide, you'll have a multi-step escalation chain that notifies progressively more people until someone acknowledges the incident.

<Accordion title="Prerequisites">
  * At least two alert channels created — see [Alerting guide](/guides/alerting)
  * A notification policy — see [First alert](/guides/first-alert)
</Accordion>

For conceptual background, see [Escalation chains](/alerting/escalation-chains).

## Build a three-tier chain

This example implements a common escalation pattern: team Slack → on-call PagerDuty → management email.

<Steps>
  <Step title="Create the alert channels">
    ```bash theme={null}
    devhelm alert-channels create \
      --name "Team Slack" --type slack \
      --webhook-url "$SLACK_WEBHOOK_URL"

    devhelm alert-channels create \
      --name "On-Call PagerDuty" --type pagerduty \
      --config "{\"routingKey\": \"$PAGERDUTY_ROUTING_KEY\"}"

    devhelm alert-channels create \
      --name "Management Email" --type email \
      --config '{"recipients": ["eng-leads@example.com"]}'
    ```
  </Step>

  <Step title="Create the notification policy">
    ```bash theme={null}
    curl -X POST https://api.devhelm.io/api/v1/notification-policies \
      -H "Authorization: Bearer $DEVHELM_API_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Critical escalation",
        "priority": 100,
        "matchRules": [
          { "type": "severity_gte", "value": "DOWN" }
        ],
        "escalation": {
          "steps": [
            {
              "delayMinutes": 0,
              "channelIds": ["<team-slack-id>"],
              "requireAck": true
            },
            {
              "delayMinutes": 10,
              "channelIds": ["<oncall-pagerduty-id>"],
              "requireAck": true,
              "repeatIntervalSeconds": 300
            },
            {
              "delayMinutes": 30,
              "channelIds": ["<management-email-id>"]
            }
          ],
          "onResolve": "notify_all_steps",
          "onReopen": "restart_from_beginning"
        }
      }'
    ```
  </Step>

  <Step title="Verify the escalation">
    Test with a failing monitor:

    ```bash theme={null}
    devhelm monitors create \
      --name "Escalation Test (delete me)" \
      --type HTTP \
      --url https://httpstat.us/500 \
      --frequency 30 \
      --regions us-east
    ```

    Watch for:

    1. Slack notification arrives immediately
    2. If no one acknowledges within 10 minutes → PagerDuty pages on-call
    3. PagerDuty repeats every 5 minutes until acknowledged
    4. If still unacknowledged at 30 minutes → management email

    Delete the test monitor when done.
  </Step>
</Steps>

## How the chain executes

```
t=0     → Step 1: Slack (requireAck: true)
         ↓ wait for ack...
t=10min → Step 2: PagerDuty (requireAck: true, repeat every 5min)
         ↓ wait for ack...
t=30min → Step 3: Email (final step)
```

If someone acknowledges at any step, the chain **stops escalating**. The incident stays open until resolved, but no further steps execute.

## Acknowledgment options

| Method             | How                                                     |
| ------------------ | ------------------------------------------------------- |
| PagerDuty/OpsGenie | Acknowledge the alert in the external system            |
| DevHelm API        | `POST /api/v1/notification-dispatches/<id>/acknowledge` |

## Resolution behavior

The `onResolve` field controls what happens when the incident resolves:

| Setting               | Behavior                                                    |
| --------------------- | ----------------------------------------------------------- |
| `notify_all_steps`    | All notified steps get a resolution message                 |
| `notify_current_step` | Only the active step gets notified                          |
| `silent`              | No resolution message (PagerDuty/OpsGenie still auto-close) |

## Variations

<AccordionGroup>
  <Accordion title="Two-tier: Slack then PagerDuty">
    ```json theme={null}
    {
      "steps": [
        { "delayMinutes": 0, "channelIds": ["<slack-id>"], "requireAck": true },
        { "delayMinutes": 15, "channelIds": ["<pagerduty-id>"], "requireAck": true, "repeatIntervalSeconds": 300 }
      ],
      "onResolve": "notify_all_steps"
    }
    ```
  </Accordion>

  <Accordion title="Immediate multi-channel (no escalation)">
    ```json theme={null}
    {
      "steps": [
        { "delayMinutes": 0, "channelIds": ["<slack-id>", "<email-id>"] }
      ],
      "onResolve": "notify_all_steps"
    }
    ```
  </Accordion>

  <Accordion title="Time-of-day routing">
    Create two notification policies at the same priority — one matching business-hours tags and one matching after-hours. The escalation chains can differ (e.g., Slack-only during the day, PagerDuty after hours).
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Escalation chains reference" icon="arrow-up-right-dots" href="/alerting/escalation-chains">
    Full step configuration and behavior options.
  </Card>

  <Card title="Alert routing by tag" icon="tags" href="/guides/alert-routing-by-tag">
    Route different monitors to different escalation chains.
  </Card>

  <Card title="Testing your alerts" icon="vial" href="/guides/testing-your-alerts">
    Validate the full escalation pipeline.
  </Card>
</CardGroup>