Test webhooks and email in CI

Give each CI job its own webhook inbox, give each test its own path and email address, and delete what the job created before it exits.

Prerequisites

A test suite in Vitest, Playwright, or pytest
A system under test whose webhook URL and recipient address you can set per test
A DevHelm API key and workspace ID:

The testing API needs an API key and a workspace ID. Export the key, look up the workspace, then export its ID:

export DEVHELM_API_TOKEN=dh_live_xxxxxxxx
curl https://api.devhelm.io/api/v1/workspaces -H "Authorization: Bearer $DEVHELM_API_TOKEN"
export DEVHELM_WORKSPACE_ID=<workspace-id>

The CLI and Python SDK read both variables. The TypeScript SDK reads DEVHELM_WORKSPACE_ID and takes the key in its constructor. With curl, send the workspace as a header:

-H "Authorization: Bearer $DEVHELM_API_TOKEN" \
-H "x-phelm-workspace-id: $DEVHELM_WORKSPACE_ID"

Plan the isolation

Parallel jobs that share an inbox see each other’s events, and an old event can satisfy a new test’s wait. Isolate at the level each resource allows:

Resource	One per	Why
Webhook inbox	CI job	Inboxes are capped per organization: 3 on Free, 10 on Developer
Path on the inbox URL	Test	Wait matches the path with `pathPrefix`
Email address	Test	Any local part is an inbox, and addresses have no cap
Email domain	Organization	Every run shares the assigned domain. Free and Developer allow one

Inboxes in use at once are matrix shards × concurrent workflow runs, plus any that developers keep. Over the cap, the create step fails with 403 INBOUND_WEBHOOK_INBOX_LIMIT_REACHED.

Store the secrets

Create an API key used only by CI, so you can revoke it on its own (see API keys). Add it and the workspace ID as repository secrets:

gh secret set DEVHELM_API_TOKEN
gh secret set DEVHELM_WORKSPACE_ID

Set DEVHELM_WORKSPACE_ID even with a single workspace, and pass both on every step that calls DevHelm. Without it the CLI and SDKs call workspace 1 and fail with 404 “Workspace doesn’t exist”.

GitHub does not pass secrets to pull_request runs from forks or Dependabot. The workflow below skips the job on fork PRs. Do not switch to pull_request_target to get secrets into those runs: checking out the fork’s code there lets that code read the key.

The workflow

Create .github/workflows/e2e.yml:

name: e2e
on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    # Fork PRs get no secrets. Skip instead of failing.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm install -g devhelm

      - name: Create webhook inbox
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: |
          devhelm inboxes create \
            --name "ci-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.shard }}" \
            -o json > inbox.json
          url=$(jq -r .httpUrl inbox.json)
          echo "::add-mask::$url"
          echo "WEBHOOK_URL=$url" >> "$GITHUB_ENV"
          echo "DEVHELM_INBOX_ID=$(jq -r .id inbox.json)" >> "$GITHUB_ENV"
          rm inbox.json

      - name: Test
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: npx vitest run --shard=${{ matrix.shard }}/2

      - name: Delete webhook inbox
        if: always() && env.DEVHELM_INBOX_ID != ''
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: devhelm inboxes delete "$DEVHELM_INBOX_ID" --yes

The ci- name prefix with run ID, attempt, and shard lets a sweeper find leftovers.
The URL is masked before it reaches $GITHUB_ENV: anyone with it can write to the inbox, and logs on public repositories are public.
The key is set per step, so npm ci and its install scripts never see it.
if: always() runs the delete when tests fail or the run is cancelled. Deleting an inbox also deletes its events and raw bodies.

For Playwright, run npx playwright test --shard=${{ matrix.shard }}/2; for pytest, pytest or pytest -n auto. The create and delete steps stay the same.

Write the tests

Each test builds its own path, registers <inbox URL><path> with the system under test, takes since before the trigger, and waits with pathPrefix. app-client stands for your own code.

import { Devhelm } from "@devhelm/sdk";
import { expect, test } from "vitest";
import { createSubscription, placeOrder } from "./app-client";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
const inboxId = process.env.DEVHELM_INBOX_ID!;
const webhookUrl = process.env.WEBHOOK_URL!;
const runId = process.env.GITHUB_RUN_ID ?? "local";

test("placing an order sends order.created", async () => {
  const path = `/${runId}/${crypto.randomUUID()}`;
  await createSubscription({ url: webhookUrl + path, events: ["order.created"] });

  const since = new Date().toISOString();
  const order = await placeOrder({ sku: "sku-123", quantity: 2 });

  const event = await client.inboxes.wait(inboxId, {
    receivedAfter: since,
    timeoutMs: 30_000,
    http: { method: "POST", pathPrefix: path },
  });
  expect(event.json()).toMatchObject({ type: "order.created", data: { id: order.id } });
}, 45_000);

import os
import uuid
from datetime import datetime, timezone
from typing import Any, cast

import pytest
from devhelm import Devhelm

from app_client import create_subscription, place_order

client = Devhelm()
INBOX_ID = os.environ["DEVHELM_INBOX_ID"]
WEBHOOK_URL = os.environ["WEBHOOK_URL"]
RUN_ID = os.environ.get("GITHUB_RUN_ID", "local")


@pytest.mark.timeout(45)
def test_placing_an_order_sends_order_created() -> None:
    path = f"/{RUN_ID}/{uuid.uuid4()}"
    create_subscription(url=WEBHOOK_URL + path, events=["order.created"])

    since = datetime.now(timezone.utc)
    order = place_order(sku="sku-123", quantity=2)

    event = client.inboxes.wait(
        INBOX_ID,
        received_after=since,
        timeout_ms=30_000,
        http={"method": "POST", "pathPrefix": path},
    )
    body = cast(dict[str, Any], event.json())
    assert body["type"] == "order.created"
    assert body["data"]["id"] == order["id"]

Set the runner timeout above timeoutMs. A longer runner timeout does not extend the HTTP call, and a shorter one kills the test before DevHelm answers. In pytest the timeout marker needs pytest-timeout.

Email: one address per test

address() returns a fresh address on the workspace’s assigned domain. Clear it when the test ends. There is no DevHelm Playwright package: a Playwright test calls the SDK the same way a Vitest test does.

signup.spec.ts

import { Devhelm, type EmailMessageDto } from "@devhelm/sdk";
import { expect, test } from "@playwright/test";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });

test("sign-up sends a 6-digit code", async ({ page }) => {
  test.setTimeout(90_000); // above the 60 s wait
  const address = await client.email.address({ label: "ci-signup" });
  try {
    await page.goto("https://staging.example.com/signup");
    await page.getByLabel("Email").fill(address.email);
    const since = new Date().toISOString();
    await page.getByRole("button", { name: "Create account" }).click();

    const message = await address.wait({ receivedAfter: since, timeoutMs: 60_000 });
    // The 1.9.0 types need this cast.
    const mail = message as unknown as EmailMessageDto & { text?: string | null; html?: string | null };
    expect(mail.otp?.[0]?.value).toMatch(/^\d{6}$/);
  } finally {
    await address.clear();
  }
});

In pytest, make the address a fixture that yields client.email.address(label=...) and calls clear() after the test. When CI has no mail provider, a test can inject the message with address.receive() instead; both paths run the same parsing. The wait response has otp and links but null text and html. The full flow, including links, is in Test a sign-up email flow.

Teardown

Resource	Who removes it	How
Webhook inbox	The workflow’s last step	`devhelm inboxes delete <inbox-id> --yes`
Email address	Each test	`address.clear()`, or `devhelm email clear --to <address> --yes`
Assigned email domain	Nobody	Keep it

Never delete the assigned domain in teardown. Every concurrent run uses it, the next address() call would create a domain with a new name, and on Free or Developer a second domain fails with 403 INBOUND_EMAIL_DOMAIN_LIMIT_REACHED. Retention does not clean up for you. DevHelm does not delete captured data on a schedule today, so events and messages stay until teardown removes them. See Security and data.

Sweep inboxes from killed runs

A lost or timed-out runner skips the delete step, and its inbox keeps a slot. Run a sweeper on a schedule or at the start of each job, with a cutoff longer than your longest job:

sweep-inboxes.ts

import { Devhelm } from "@devhelm/sdk";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
const cutoff = Date.now() - 60 * 60 * 1000; // 1 hour

for (const inbox of await client.inboxes.list({ search: "ci-" })) {
  // search is a case-insensitive substring match, so check the prefix again
  if (inbox.name.startsWith("ci-") && Date.parse(inbox.createdAt) < cutoff) {
    await client.inboxes.delete(inbox.id);
  }
}

Parallel runs and the wait budget

All wait calls in an organization share 40 per minute: webhook and email waits, every workspace, every CI job, and every developer running tests locally. Over the limit, a wait returns 429 RATE_LIMITED with Retry-After. A suite with 30 waits fits in one minute; two shards of it plus a second PR in flight use 120 and get throttled.

Wait once per test, then assert everything on the returned event or message.
Use one long timeoutMs (up to 120000), not a loop of short waits. Each call counts.
Limit concurrent runs with a GitHub concurrency group when the inbox cap or the wait budget is tight:

concurrency:
  group: devhelm-e2e
  cancel-in-progress: false

Back off on 429

Neither SDK retries on its own. Wrap waits in a helper that sleeps for retryAfter and tries again. Retrying with the same since is safe, because wait does not consume what it returns.

import { DevhelmRateLimitError } from "@devhelm/sdk";

export async function withWaitBudget<T>(call: () => Promise<T>, attempts = 3): Promise<T> {
  for (let attempt = 1; ; attempt++) {
    try {
      return await call();
    } catch (err) {
      if (!(err instanceof DevhelmRateLimitError) || attempt >= attempts) throw err;
      await new Promise((resolve) => setTimeout(resolve, (err.retryAfter ?? 5) * 1000));
    }
  }
}

const event = await withWaitBudget(() =>
  client.inboxes.wait(inboxId, { receivedAfter: since, timeoutMs: 30_000, http: { pathPrefix: path } }),
);

In Python, catch DevhelmRateLimitError and sleep for err.retry_after the same way. Size the runner timeout for the retries too: attempts × (timeoutMs + retryAfter).

Monthly quota

Every captured webhook request and every received or injected email counts toward one monthly event cap per organization: 300 on Free, 3,000 on Developer, 100,000 on Team. A suite that sends 30 events and runs 20 times a day needs 18,000 a month. At the cap, webhook senders and inject get 403 INBOUND_EVENTS_MONTHLY_LIMIT_REACHED and nothing is stored. The counter updates asynchronously, so a burst can go slightly over first, and an inject accepted right at the cap can be dropped, which makes its wait return 408. See Limits, quotas, and errors.

Troubleshooting

Create step fails with 403 INBOUND_WEBHOOK_INBOX_LIMIT_REACHED

The organization is at its inbox cap, counting every workspace. Delete leftovers from killed runs (the sweeper does this), reduce matrix shards, or add a concurrency group.

Wait returns 408 in CI but passes locally

since was taken after the trigger, or not passed, and a fast event arrived before the wait started.
The test’s pathPrefix does not match the path the system under test called. Paths are case-sensitive.
The system under test is slower in CI. Raise timeoutMs and the runner timeout with it.
The monthly cap was reached. Check the sender’s response for 403.

The job is skipped on pull requests

The PR comes from a fork, and fork runs get no secrets. Run the suite after merge, or have a maintainer push the branch to the main repository.

Next steps

Wait for a webhook

Wait semantics, the Nth event, and negative assertions.

Wait for an email

Addresses, subjectContains, and repeat sends.

Limits, quotas, and errors

Every limit and error code with its fix.

Security and data

Keys in CI, captured headers, and cleanup.

Overview

Getting Started

Monitoring Guides

Alerting Guides

Testing Guides

Incident Guides

Status Data Guides

Monitoring as Code Guides

Test webhooks and email in CI

Plan the isolation

Store the secrets

The workflow

Write the tests

Email: one address per test

Teardown

Sweep inboxes from killed runs

Parallel runs and the wait budget

Back off on 429

Monthly quota

Troubleshooting

Next steps

Wait for a webhook

Wait for an email

Limits, quotas, and errors

Security and data

​Plan the isolation

​Store the secrets

​The workflow

​Write the tests

​Email: one address per test

​Teardown

​Sweep inboxes from killed runs

​Parallel runs and the wait budget

​Back off on 429

​Monthly quota

​Troubleshooting

​Next steps

Wait for a webhook

Wait for an email

Limits, quotas, and errors

Security and data

Plan the isolation

Store the secrets

The workflow

Write the tests

Email: one address per test

Teardown

Sweep inboxes from killed runs

Parallel runs and the wait budget

Back off on 429

Monthly quota

Troubleshooting

Next steps