> ## Documentation Index
> Fetch the complete documentation index at: https://docs.devhelm.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Test webhooks and email in CI

> Run webhook and email tests in GitHub Actions with one inbox per job, a path and address per test, teardown, and a shared wait budget

Give each CI job its own webhook inbox, give each test its own path and email address, and delete what the job created before it exits.

<Accordion title="Prerequisites">
  * A test suite in Vitest, Playwright, or pytest
  * A system under test whose webhook URL and recipient address you can set per test
  * A DevHelm API key and workspace ID:

  The testing API needs an API key and a workspace ID. Export the key, look up the workspace, then export its ID:

  ```bash theme={null}
  export DEVHELM_API_TOKEN=dh_live_xxxxxxxx
  curl https://api.devhelm.io/api/v1/workspaces -H "Authorization: Bearer $DEVHELM_API_TOKEN"
  export DEVHELM_WORKSPACE_ID=<workspace-id>
  ```

  The CLI and Python SDK read both variables. The TypeScript SDK reads `DEVHELM_WORKSPACE_ID` and takes the key in its constructor. With curl, send the workspace as a header:

  ```bash theme={null}
  -H "Authorization: Bearer $DEVHELM_API_TOKEN" \
  -H "x-phelm-workspace-id: $DEVHELM_WORKSPACE_ID"
  ```
</Accordion>

## Plan the isolation

Parallel jobs that share an inbox see each other's events, and an old event can satisfy a new test's wait. Isolate at the level each resource allows:

| Resource | One per | Why |
| - | - | - |
| Webhook inbox | CI job | Inboxes are capped per organization: 3 on Free, 10 on Developer |
| Path on the inbox URL | Test | Wait matches the path with `pathPrefix` |
| Email address | Test | Any local part is an inbox, and addresses have no cap |
| Email domain | Organization | Every run shares the assigned domain. Free and Developer allow one |

Inboxes in use at once are matrix shards × concurrent workflow runs, plus any that developers keep. Over the cap, the create step fails with `403 INBOUND_WEBHOOK_INBOX_LIMIT_REACHED`.

## Store the secrets

Create an API key used only by CI, so you can revoke it on its own (see [API keys](/platform/api-keys)). Add it and the workspace ID as repository secrets:

```bash theme={null}
gh secret set DEVHELM_API_TOKEN
gh secret set DEVHELM_WORKSPACE_ID
```

Set `DEVHELM_WORKSPACE_ID` even with a single workspace, and pass both on every step that calls DevHelm. Without it the CLI and SDKs call workspace `1` and fail with `404` "Workspace doesn't exist".

<Warning>
  GitHub does not pass secrets to `pull_request` runs from forks or Dependabot. The workflow below skips the job on fork PRs. Do not switch to `pull_request_target` to get secrets into those runs: checking out the fork's code there lets that code read the key.
</Warning>

## The workflow

Create `.github/workflows/e2e.yml`:

```yaml theme={null}
name: e2e
on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    # Fork PRs get no secrets. Skip instead of failing.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm install -g devhelm

      - name: Create webhook inbox
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: |
          devhelm inboxes create \
            --name "ci-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.shard }}" \
            -o json > inbox.json
          url=$(jq -r .httpUrl inbox.json)
          echo "::add-mask::$url"
          echo "WEBHOOK_URL=$url" >> "$GITHUB_ENV"
          echo "DEVHELM_INBOX_ID=$(jq -r .id inbox.json)" >> "$GITHUB_ENV"
          rm inbox.json

      - name: Test
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: npx vitest run --shard=${{ matrix.shard }}/2

      - name: Delete webhook inbox
        if: always() && env.DEVHELM_INBOX_ID != ''
        env:
          DEVHELM_API_TOKEN: ${{ secrets.DEVHELM_API_TOKEN }}
          DEVHELM_WORKSPACE_ID: ${{ secrets.DEVHELM_WORKSPACE_ID }}
        run: devhelm inboxes delete "$DEVHELM_INBOX_ID" --yes
```

* The `ci-` name prefix with run ID, attempt, and shard lets a sweeper find leftovers.
* The URL is masked before it reaches `$GITHUB_ENV`: anyone with it can write to the inbox, and logs on public repositories are public.
* The key is set per step, so `npm ci` and its install scripts never see it.
* `if: always()` runs the delete when tests fail or the run is cancelled. Deleting an inbox also deletes its events and raw bodies.

For Playwright, run `npx playwright test --shard=${{ matrix.shard }}/2`; for pytest, `pytest` or `pytest -n auto`. The create and delete steps stay the same.

## Write the tests

Each test builds its own path, registers `<inbox URL><path>` with the system under test, takes `since` before the trigger, and waits with `pathPrefix`. `app-client` stands for your own code.

<CodeGroup>
  ```typescript order-webhooks.test.ts theme={null}
  import { Devhelm } from "@devhelm/sdk";
  import { expect, test } from "vitest";
  import { createSubscription, placeOrder } from "./app-client";

  const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
  const inboxId = process.env.DEVHELM_INBOX_ID!;
  const webhookUrl = process.env.WEBHOOK_URL!;
  const runId = process.env.GITHUB_RUN_ID ?? "local";

  test("placing an order sends order.created", async () => {
    const path = `/${runId}/${crypto.randomUUID()}`;
    await createSubscription({ url: webhookUrl + path, events: ["order.created"] });

    const since = new Date().toISOString();
    const order = await placeOrder({ sku: "sku-123", quantity: 2 });

    const event = await client.inboxes.wait(inboxId, {
      receivedAfter: since,
      timeoutMs: 30_000,
      http: { method: "POST", pathPrefix: path },
    });
    expect(event.json()).toMatchObject({ type: "order.created", data: { id: order.id } });
  }, 45_000);
  ```

  ```python test_order_webhooks.py theme={null}
  import os
  import uuid
  from datetime import datetime, timezone
  from typing import Any, cast

  import pytest
  from devhelm import Devhelm

  from app_client import create_subscription, place_order

  client = Devhelm()
  INBOX_ID = os.environ["DEVHELM_INBOX_ID"]
  WEBHOOK_URL = os.environ["WEBHOOK_URL"]
  RUN_ID = os.environ.get("GITHUB_RUN_ID", "local")


  @pytest.mark.timeout(45)
  def test_placing_an_order_sends_order_created() -> None:
      path = f"/{RUN_ID}/{uuid.uuid4()}"
      create_subscription(url=WEBHOOK_URL + path, events=["order.created"])

      since = datetime.now(timezone.utc)
      order = place_order(sku="sku-123", quantity=2)

      event = client.inboxes.wait(
          INBOX_ID,
          received_after=since,
          timeout_ms=30_000,
          http={"method": "POST", "pathPrefix": path},
      )
      body = cast(dict[str, Any], event.json())
      assert body["type"] == "order.created"
      assert body["data"]["id"] == order["id"]
  ```
</CodeGroup>

Set the runner timeout above `timeoutMs`. A longer runner timeout does not extend the HTTP call, and a shorter one kills the test before DevHelm answers. In pytest the timeout marker needs `pytest-timeout`.

### Email: one address per test

`address()` returns a fresh address on the workspace's assigned domain. Clear it when the test ends. There is no DevHelm Playwright package: a Playwright test calls the SDK the same way a Vitest test does.

```typescript signup.spec.ts theme={null}
import { Devhelm, type EmailMessageDto } from "@devhelm/sdk";
import { expect, test } from "@playwright/test";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });

test("sign-up sends a 6-digit code", async ({ page }) => {
  test.setTimeout(90_000); // above the 60 s wait
  const address = await client.email.address({ label: "ci-signup" });
  try {
    await page.goto("https://staging.example.com/signup");
    await page.getByLabel("Email").fill(address.email);
    const since = new Date().toISOString();
    await page.getByRole("button", { name: "Create account" }).click();

    const message = await address.wait({ receivedAfter: since, timeoutMs: 60_000 });
    // The 1.9.0 types need this cast.
    const mail = message as unknown as EmailMessageDto & { text?: string | null; html?: string | null };
    expect(mail.otp?.[0]?.value).toMatch(/^\d{6}$/);
  } finally {
    await address.clear();
  }
});
```

In pytest, make the address a fixture that yields `client.email.address(label=...)` and calls `clear()` after the test. When CI has no mail provider, a test can inject the message with `address.receive()` instead; both paths run the same parsing. The wait response has `otp` and `links` but `null` `text` and `html`. The full flow, including links, is in [Test a sign-up email flow](/guides/test-signup-email-flow).

## Teardown

| Resource | Who removes it | How |
| - | - | - |
| Webhook inbox | The workflow's last step | `devhelm inboxes delete <inbox-id> --yes` |
| Email address | Each test | `address.clear()`, or `devhelm email clear --to <address> --yes` |
| Assigned email domain | Nobody | Keep it |

Never delete the assigned domain in teardown. Every concurrent run uses it, the next `address()` call would create a domain with a new name, and on Free or Developer a second domain fails with `403 INBOUND_EMAIL_DOMAIN_LIMIT_REACHED`.

Retention does not clean up for you. DevHelm does not delete captured data on a schedule today, so events and messages stay until teardown removes them. See [Security and data](/testing/security#retention-and-cleanup).

### Sweep inboxes from killed runs

A lost or timed-out runner skips the delete step, and its inbox keeps a slot. Run a sweeper on a schedule or at the start of each job, with a cutoff longer than your longest job:

```typescript sweep-inboxes.ts theme={null}
import { Devhelm } from "@devhelm/sdk";

const client = new Devhelm({ token: process.env.DEVHELM_API_TOKEN! });
const cutoff = Date.now() - 60 * 60 * 1000; // 1 hour

for (const inbox of await client.inboxes.list({ search: "ci-" })) {
  // search is a case-insensitive substring match, so check the prefix again
  if (inbox.name.startsWith("ci-") && Date.parse(inbox.createdAt) < cutoff) {
    await client.inboxes.delete(inbox.id);
  }
}
```

## Parallel runs and the wait budget

All wait calls in an organization share 40 per minute: webhook and email waits, every workspace, every CI job, and every developer running tests locally. Over the limit, a wait returns `429 RATE_LIMITED` with `Retry-After`. A suite with 30 waits fits in one minute; two shards of it plus a second PR in flight use 120 and get throttled.

* Wait once per test, then assert everything on the returned event or message.
* Use one long `timeoutMs` (up to 120000), not a loop of short waits. Each call counts.
* Limit concurrent runs with a GitHub `concurrency` group when the inbox cap or the wait budget is tight:

```yaml theme={null}
concurrency:
  group: devhelm-e2e
  cancel-in-progress: false
```

### Back off on 429

Neither SDK retries on its own. Wrap waits in a helper that sleeps for `retryAfter` and tries again. Retrying with the same `since` is safe, because wait does not consume what it returns.

```typescript theme={null}
import { DevhelmRateLimitError } from "@devhelm/sdk";

export async function withWaitBudget<T>(call: () => Promise<T>, attempts = 3): Promise<T> {
  for (let attempt = 1; ; attempt++) {
    try {
      return await call();
    } catch (err) {
      if (!(err instanceof DevhelmRateLimitError) || attempt >= attempts) throw err;
      await new Promise((resolve) => setTimeout(resolve, (err.retryAfter ?? 5) * 1000));
    }
  }
}

const event = await withWaitBudget(() =>
  client.inboxes.wait(inboxId, { receivedAfter: since, timeoutMs: 30_000, http: { pathPrefix: path } }),
);
```

In Python, catch `DevhelmRateLimitError` and sleep for `err.retry_after` the same way. Size the runner timeout for the retries too: `attempts × (timeoutMs + retryAfter)`.

## Monthly quota

Every captured webhook request and every received or injected email counts toward one monthly event cap per organization: 300 on Free, 3,000 on Developer, 100,000 on Team. A suite that sends 30 events and runs 20 times a day needs 18,000 a month.

At the cap, webhook senders and inject get `403 INBOUND_EVENTS_MONTHLY_LIMIT_REACHED` and nothing is stored. The counter updates asynchronously, so a burst can go slightly over first, and an inject accepted right at the cap can be dropped, which makes its wait return `408`. See [Limits, quotas, and errors](/testing/limits).

## Troubleshooting

<AccordionGroup>
  <Accordion title="Create step fails with 403 INBOUND_WEBHOOK_INBOX_LIMIT_REACHED">
    The organization is at its inbox cap, counting every workspace. Delete leftovers from killed runs (the sweeper does this), reduce matrix shards, or add a `concurrency` group.
  </Accordion>

  <Accordion title="Wait returns 408 in CI but passes locally">
    * `since` was taken after the trigger, or not passed, and a fast event arrived before the wait started.
    * The test's `pathPrefix` does not match the path the system under test called. Paths are case-sensitive.
    * The system under test is slower in CI. Raise `timeoutMs` and the runner timeout with it.
    * The monthly cap was reached. Check the sender's response for `403`.
  </Accordion>

  <Accordion title="The job is skipped on pull requests">
    The PR comes from a fork, and fork runs get no secrets. Run the suite after merge, or have a maintainer push the branch to the main repository.
  </Accordion>
</AccordionGroup>

## Next steps

<CardGroup cols={2}>
  <Card title="Wait for a webhook" icon="code" href="/testing/webhooks/assert-in-tests">
    Wait semantics, the Nth event, and negative assertions.
  </Card>

  <Card title="Wait for an email" icon="envelope" href="/testing/email/wait">
    Addresses, `subjectContains`, and repeat sends.
  </Card>

  <Card title="Limits, quotas, and errors" icon="gauge" href="/testing/limits">
    Every limit and error code with its fix.
  </Card>

  <Card title="Security and data" icon="shield" href="/testing/security">
    Keys in CI, captured headers, and cleanup.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.