Alerting Guide

This guide walks you through configuring the alerting system end to end. For conceptual background, see Alerting overview.

Step 1: Create alert channels

Alert channels are the destinations where notifications get sent. Create them first, then reference them in notification policies.

# Slack
devhelm alert-channels create \
  --name "Engineering Slack" \
  --type slack \
  --webhook-url "https://hooks.slack.com/services/T.../B.../xxx"

# PagerDuty
devhelm alert-channels create \
  --name "On-Call PagerDuty" \
  --type pagerduty \
  --config '{"channelType":"pagerduty","routingKey":"your-events-api-v2-routing-key"}'

# Email
devhelm alert-channels create \
  --name "Team Email" \
  --type email \
  --config '{"channelType":"email","recipients":["oncall@example.com","team@example.com"]}'

alertChannels:
  - name: Engineering Slack
    config:
      channelType: slack
      webhookUrl: ${SLACK_WEBHOOK_URL}

  - name: On-Call PagerDuty
    config:
      channelType: pagerduty
      routingKey: ${PAGERDUTY_ROUTING_KEY}

  - name: Team Email
    config:
      channelType: email
      recipients:
        - oncall@example.com
        - team@example.com

Test a channel

Verify your channel is configured correctly:

devhelm alert-channels test <channel-id>

This sends a test notification through the channel and reports success or failure.

Step 2: Create notification policies

Notification policies connect incidents to alert channels through match rules and escalation chains.

Simple policy (notify Slack for all incidents)

curl -X POST https://api.devhelm.io/api/v1/notification-policies \
  -H "Authorization: Bearer $DEVHELM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "All incidents to Slack",
    "matchRules": [],
    "escalation": {
      "steps": [
        {
          "delayMinutes": 0,
          "channelIds": ["<slack-channel-id>"]
        }
      ]
    },
    "priority": 0
  }'

An empty matchRules array is a catch-all — it matches every incident.

Tiered escalation

Route critical incidents through an escalation chain:

{
  "name": "Critical escalation",
  "matchRules": [
    { "type": "severity_gte", "value": "DOWN" }
  ],
  "escalation": {
    "steps": [
      {
        "delayMinutes": 0,
        "channelIds": ["<slack-channel-id>"],
        "requireAck": false
      },
      {
        "delayMinutes": 15,
        "channelIds": ["<pagerduty-channel-id>"],
        "requireAck": true,
        "repeatIntervalSeconds": 300
      },
      {
        "delayMinutes": 30,
        "channelIds": ["<email-channel-id>"]
      }
    ],
    "onResolve": "notify_all_steps",
    "onReopen": "restart_from_beginning"
  },
  "priority": 10
}

This policy:

Immediately notifies Slack
After 15 minutes (if not acknowledged), pages PagerDuty and repeats every 5 minutes
After 30 minutes, emails the team

Scoped policies

Use match rules to target specific monitors or conditions:

{
  "name": "Production HTTP monitors only",
  "matchRules": [
    { "type": "monitor_type_in", "values": ["HTTP"] },
    { "type": "monitor_tag_in", "values": ["production"] }
  ],
  "escalation": {
    "steps": [
      { "delayMinutes": 0, "channelIds": ["<channel-id>"] }
    ]
  },
  "priority": 5
}

Available match rules

Rule	Matches on	Fields
`severity_gte`	Severity ≥ value	`value`: `DOWN`, `DEGRADED`, `MAINTENANCE`
`monitor_id_in`	Specific monitors	`monitorIds`: list of UUIDs
`monitor_type_in`	Monitor types	`values`: `HTTP`, `TCP`, `DNS`, etc.
`monitor_tag_in`	Monitor tags	`values`: tag names
`region_in`	Affected regions	`regions`: region codes
`incident_status`	Event type	`value`: `created`, `resolved`, `reopened`
`service_id_in`	Status data services	`values`: service IDs
`resource_group_id_in`	Resource groups	`values`: group IDs
`component_name_in`	Status data component names	`values`: component names

Step 3: Set policy priority

Policies are evaluated by priority (highest first). All matching policies execute — there’s no “first match wins”. Set higher priority on more specific policies:

Priority	Policy	Match rules
10	Critical escalation	`severity_gte: DOWN`
5	Production HTTP to Slack	`monitor_type_in: HTTP` + `monitor_tag_in: production`
0	Everything to email	(catch-all)

Testing your setup

Test individual channels: devhelm alert-channels test <id>
Create a test monitor with a low frequency that will fail:

devhelm monitors create \
  --name "Alert Test" \
  --type HTTP \
  --url https://httpstat.us/500 \
  --frequency 300 \
  --regions us-east

Wait for the incident to trigger and verify notifications arrive
Delete the test monitor when done

Next steps

Integration setup

Detailed setup guides for each alert channel type.

Incidents guide

Manage incidents and maintenance windows.

Overview

Getting Started

Monitoring Guides

Alerting Guides

Incident Guides

Status Data Guides

Monitoring as Code Guides

Step 1: Create alert channels

Test a channel

Step 2: Create notification policies

Simple policy (notify Slack for all incidents)

Tiered escalation

Scoped policies

Available match rules

Step 3: Set policy priority

Testing your setup

Next steps

Integration setup

Incidents guide

​Step 1: Create alert channels

​Test a channel

​Step 2: Create notification policies

​Simple policy (notify Slack for all incidents)

​Tiered escalation

​Scoped policies

​Available match rules

​Step 3: Set policy priority

​Testing your setup

​Next steps

Integration setup

Incidents guide

Step 1: Create alert channels

Test a channel

Step 2: Create notification policies

Simple policy (notify Slack for all incidents)

Tiered escalation

Scoped policies

Available match rules

Step 3: Set policy priority

Testing your setup

Next steps