crashmire-results/1
An open format for AI agent test results
Every CrashMire crash test comes with its results in an open JSON format: what was tested, on which state, how often and with what uncertainty, and which regulatory references. Anyone is free to read it, produce it or adopt it.
Why an open format
- A buyer, an insurer or a compliance tool compares results without reading a 40-page PDF report.
- The results stay yours: put them in your own risk register, your AI Act file or your client's.
- The attack method stays private: the format describes results, never the content of the trap messages.
The fields
| Field | What it holds |
|---|---|
format | The format version, today crashmire-results/1. |
report | The report reference. |
tested_at | The crash test date. |
method | The method version (the CrashMire standard). |
state | The tested state: its fingerprint (instructions, tools, models, tests), the models and tests played. |
scope | The number of runs, models and turns, and the number of runs where the agent gave in. |
verdict | The overall level: none, low, medium, high or critical. |
by_severity | The number of flaws per level. |
findings[] | Each flaw: its kind, the test, the tool involved, the runs where the agent gave in out of those played, and the detail per model. |
findings[].range_95 | The 95 % range of the frequency (in %), and indicative = true under 10 runs. |
findings[].references | The OWASP LLM / agentic, AI Act and other framework references concerned. |
loss_ceiling | For an agent that moves money: the largest amount granted outside its rules in one conversation, and in how many runs. |
not_a_certification | Always true: a test result is not a certification. |
Example
{
"format": "crashmire-results/1",
"report": "CM-2026-0042",
"tested_at": "2026-10-20",
"method": "v1.1",
"state": { "fingerprint": "a7cde91fa2f6876d", "models": ["…"], "tests": ["pressure", "injection", "authority"] },
"scope": { "runs": 90, "models": 3, "turns": 1080, "runs_gave_in": 21 },
"verdict": "high",
"by_severity": { "critical": 0, "high": 2, "medium": 1, "low": 0 },
"findings": [{
"id": "F-01", "kind": "pressure", "test": "pressure", "severity": "high", "tool": "refund",
"gave_in": 6, "runs": 30, "range_95": [10, 38], "indicative": false,
"models": [{ "model": "…", "gave_in": 5, "runs": 10 }],
"references": { "owasp_llm": ["LLM06"], "ai_act": ["Art. 15"] }
}],
"loss_ceiling": { "max": 340, "euro": true, "runs": 4, "of": 90 },
"not_a_certification": true
}The rules
- The format is free to reuse (CC BY 4.0): cite "crashmire-results", that is all.
- A new version adds fields, it never removes any; the format field says which one you read.
- A result holds only for the state described by its fingerprint: any change of the agent calls for a new test.
- No field contains the text of the attacks or any personal data.