Automatic re-test
Tested once does not mean tested forever.
A tweaked prompt, a new tool, a model updated by its vendor: every change can reopen a closed flaw. CrashMire re-tests the agent on every change and tells you what regressed, before your customers do.
What makes an agent regress
- An instruction added for one customer case, which weakens another.
- A new tool: the agent now has a button it can press wrongly.
- A model changed, or silently updated by its vendor: same prompt, different behaviour.
- New data (a knowledge base, a channel) where an instruction can hide.
Every month, hands off
With the annual plan, every agent is crash-tested again each month. The report says what improved, what regressed, and alerts you on a new flaw.
From n8n, on every change
Import our workflow template: as soon as you change the agent's workflow, it starts the crash test, waits for the verdict and tells you if there is a new flaw. Your key stays in n8n's credential manager.
Before every release
One command to add to your delivery pipeline: it starts the agent's crash test, waits for the verdict and blocks the release on a regression.
In your delivery pipeline
One line is enough, in GitHub Actions or GitLab CI for example. The exit code is 0 if nothing regressed, 1 otherwise: the release stops on its own.
# .github/workflows/release.yml
- name: CrashMire
env:
CRASHMIRE_KEY: ${{ secrets.CRASHMIRE_KEY }}
run: ./crashmire-gate.sh 12 "${{ github.sha }}" newWhat the verdict says
- New flaws: absent from the previous test, they block the release (by default).
- The flaws fixed and those remaining, with their frequency and its range.
- The fingerprint of the tested state: instructions, tools, models. You know exactly which version was judged.
- A threshold of your choice: block on a new flaw, on any critical or high flaw, or on any flaw.
The release check is an option of the annual plan and of the integrator plan; the key is created in your client space. See pricing