> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devhelm.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Splunk On-Call

> Create and auto-resolve Splunk On-Call incidents from DevHelm monitor failures

Connect DevHelm to Splunk On-Call (formerly VictorOps) to automatically open and resolve incidents. DevHelm posts to the Splunk On-Call **REST endpoint integration**, so incidents follow your existing escalation policies and on-call rotations.

## Setup

<Steps>
  <Step title="Get your REST endpoint API key and routing key">
    1. In Splunk On-Call, go to **Settings** → **Integrations** → **REST Endpoint**
    2. Enable the integration and copy the **API key** from the endpoint URL
    3. Go to **Settings** → **Routing Keys** and copy an existing key — or create a new one (e.g. `devhelm`) mapped to the team that should receive alerts
  </Step>

  <Step title="Create the alert channel in DevHelm">
    <CodeGroup>
      ```bash CLI theme={null}
      devhelm alert-channels create \
        --name "Splunk On-Call" \
        --type splunk_oncall \
        --config '{"channelType":"splunk_oncall","apiKey":"your-rest-endpoint-api-key","routingKey":"devhelm"}'
      ```

      ```yaml devhelm.yml theme={null}
      alertChannels:
        - name: Splunk On-Call
          config:
            channelType: splunk_oncall
            apiKey: ${SPLUNK_ONCALL_API_KEY}
            routingKey: ${SPLUNK_ONCALL_ROUTING_KEY}
      ```
    </CodeGroup>
  </Step>

  <Step title="Test the channel">
    ```bash theme={null}
    devhelm alert-channels test <channel-id>
    ```
  </Step>
</Steps>

## Configuration

| Field        | Description                            | Required |
| ------------ | -------------------------------------- | -------- |
| `apiKey`     | Splunk On-Call REST endpoint API key   | Yes      |
| `routingKey` | Routing key that maps alerts to a team | Yes      |

## Lifecycle behavior

Splunk On-Call uses a **trigger-resolve** lifecycle keyed on the incident, so state stays in sync with DevHelm:

| DevHelm event     | Splunk On-Call action                                    |
| ----------------- | -------------------------------------------------------- |
| Incident created  | Triggers an incident that follows your escalation policy |
| Incident resolved | Auto-resolves the corresponding incident                 |
| Incident reopened | Triggers a new incident                                  |

## Troubleshooting

<AccordionGroup>
  <Accordion title="Alerts not creating (HTTP 4xx)">
    Confirm you used the **REST endpoint API key**, not a personal API token from **Settings** → **API**. The two are different credentials — only the REST endpoint key works with this integration. Then run `devhelm alert-channels test <channel-id>`.
  </Accordion>

  <Accordion title="Incidents created but no one is paged">
    The routing key must be mapped to a team with an active escalation policy. In **Settings** → **Routing Keys**, verify the key points to the right team, then check that team has an on-call schedule.
  </Accordion>
</AccordionGroup>