The crash test in your delivery pipeline.

How it works

1. Buy crash tests

   Online payment, automatic invoice. Each credit is one full crash test: six attack families, several models, several runs per scenario.

2. Create your key

   In your client space. It is shown once; we keep only its fingerprint. Replace it or withdraw it at any time.

3. Run it from your code

   Describe the agent (or paste its configuration export), read the verdict, then re-test it at every change. The report also arrives by email.

Pricing

The price of the full online crash test, as prepaid credits. A credit is only used when the crash test starts; if it fails on our side, you get it back.

1 crash test 1,900 € excl. VAT To try the API on one agent. 5 crash tests 7,600 € excl. VAT −20 %: a first test and four re-tests, or five agents.

Email (your client space)

Company

Website

I accept the terms of sale

Pay and get my credits

Card payment through Stripe. Your credits are added as soon as you pay; the key is created in your client space.

Start in 3 calls

Every call is HTTPS, with the key in the Authorization header. Answers are JSON.

1. Start an agent's crash test

curl -X POST https://crashmire.com/api/v1/tests \
  -H "Authorization: Bearer $CRASHMIRE_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: first-test-001" \
  -d '{
    "description": "Customer service assistant of an online shop. It looks up orders, refunds up to €50 without approval, never shares another customer's data.",
    "locale": "en",
    "reference": "v1.4.0",
    "webhook_url": "https://example.com/hooks/crashmire"
  }'
# → 202 {"id": "ct_3f9a…", "status": "building", "status_url": "…", "credits_left": 4}

You can add the agent's export in the config field (n8n, Flowise, Dify, Make, OpenAI Assistant, MCP tool list): the rebuilt agent then sticks to its real instructions and tools.

2. Read the verdict

curl -H "Authorization: Bearer $CRASHMIRE_KEY" "https://crashmire.com/api/v1/tests/ct_3f9a…?fail_on=high"
# building → running → done
# → {"status": "done", "verdict": "fail", "by_severity": {"critical": 0, "high": 1, …},
#    "findings": [ … ], "new": [ … ], "report_url": "…", "results": { "format": "crashmire-results/1", … }}

3. Re-test it after every change

curl -X POST https://crashmire.com/api/v1/tests/ct_3f9a…/retest \
  -H "Authorization: Bearer $CRASHMIRE_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $GIT_SHA" -d '{"reference": "'$GIT_SHA'"}'

The re-test replays the same scenarios on the same agent and compares with its last result: the new field lists the flaws that appeared since. That is what blocks a release.

In your CI

A 60-line script (curl and jq) starts the re-test, waits for the verdict and sets the exit code: 0 when nothing regressed, 1 otherwise. The same commit run twice uses one credit.

Download crashmire-ci.sh

Copy it into your repository rather than downloading it at every run, and check its fingerprint: sha256 047dcfbad40d784c9e6412672643104f1f6f1feb4fb1b2c07126bd25abb2841b

GitHub Actions

# .github/workflows/crashmire.yml
name: CrashMire crash test
on:
  push:
    paths: ["prompts/**", "agent/**"]   # when the prompt, the tools or the model change
jobs:
  crash-test:
    runs-on: ubuntu-latest
    timeout-minutes: 100
    steps:
      - uses: actions/checkout@v4
      - name: Crash-test the agent
        env:
          CRASHMIRE_KEY: ${{ secrets.CRASHMIRE_KEY }}
        run: ./ci/crashmire-ci.sh ct_3f9a… new

GitLab CI

crashmire:
  stage: test
  image: alpine:3.20
  before_script: [apk add --no-cache bash curl jq]
  script: [./ci/crashmire-ci.sh ct_3f9a… new]
  rules:
    - changes: ["prompts/**", "agent/**"]
  # CRASHMIRE_KEY: a masked, protected CI/CD variable

Your agents run on n8n? The workflow template is on the Automatic re-test page

Reference

Authorization: Bearer cmk_… on every call. A key in the address (?key=) is refused: it would end up in logs.

Start fields

• description — required, 40 to 8,000 characters: what the agent does, its tools, its limits.
• config — optional: the agent's export (JSON object or text), 400 kB at most.
• locale — fr or en: the report's language (en by default).
• reference — optional, 80 characters: a commit, a version; echoed back.
• webhook_url — optional: a public https address, called at the end (signed).
• Idempotency-Key — optional header, recommended: the same request sent twice starts one crash test.

States

• building — the agent is being rebuilt from its description.
• running — the crash test is running (from minutes to an hour, depending on the agent).
• in_review — a quality check asked for a human reading before sending (usually within a working day).
• done — finished: verdict and results available.
• failed — the crash test could not complete on our side; the credit is given back.

Verdict threshold (fail_on)

• new — fails when a flaw absent from the previous test appears (default).
• high — fails on any high or critical flaw.
• critical — fails on any critical flaw.
• any — fails as soon as a flaw exists.

Errors

Webhooks

If you give webhook_url, we POST JSON to it when the crash test ends (done, failed or in_review), signed with the secret from your client space: header CrashMire-Signature: t=<timestamp>,v1=<HMAC-SHA256 of "t.body">. Check the signature before reading the body.

# Python: check the signature before trusting the body
import hmac, hashlib, time

def verify(body: bytes, header: str, secret: str) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    if abs(time.time() - int(parts["t"])) > 300:      # older than 5 minutes: replay
        return False
    expected = hmac.new(secret.encode(), parts["t"].encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"])

# verify(request.body, request.headers["CrashMire-Signature"], "whsec_…")

We only call public https addresses (checked at every call), without following redirects, within 5 seconds; an answer other than 2xx is retried three times. When in doubt, the GET call is the source of truth.

Security

• The key is shown once; we keep only its SHA-256 fingerprint. Replace or withdraw it in one click from your client space.
• Each credit is taken in a single database operation: two simultaneous calls can never spend the same credit.
• Per-account limits: 5 crash tests in progress, 20 starts per 24 hours, 60 calls a minute. A stolen key cannot empty your credits at once; we raise the limit on request.
• Addresses that send many invalid keys are blocked automatically.
• HTTPS only, key in the header only, nothing cached (Cache-Control: no-store).
• Webhooks are signed, timestamped, and only go to public addresses: never into your internal network or ours.
• The crash test runs on simulated tools: nothing is executed on your side or your clients'.
• We see every start and every report; we can suspend an account in case of abuse.

What the API never returns

Our scenarios, attack messages, prompts and the way we judge stay with us. The API returns verdicts, frequencies, severities and references (OWASP, AI Act), never the content of the attacks.

Results follow the open format crashmire-results/1.