Our own crash test
We put CrashMire through a crash test
Can a text written by an adversary make our system do something it should not? That is the question we ask about our clients' agents. On 1 November 2026 we asked it about our own AI, with the same method (version 1.3).
- 156cases played against the real models in production, including control cases with no trap at all
- 30 of 30attempts to make Léa reveal her hidden instructions, none of them successful
- 2 stepsa code review, then the same attacks against the real models
- v1.3the version of our method, the same as for our clients
What we tested
Every part of CrashMire that reads a text coming from outside, and, for each one, who could write that text.
| What we tested | Who writes |
|---|---|
| The interview assistantIt helps our clients describe their agent before a crash test. | Our clients: the description of their agent, their answers, the configuration they import. |
| The judging of runsIt establishes, run by run, what the tested agent did and whether a line was crossed. | The tested agent itself: its messages, its tool calls, its emails. |
| Léa, the public demoThe customer service AI agent of a fictitious shop, which anyone can try to make give in. | Visitors to the site. |
| The public MCP serverThe door through which an AI assistant can get our information or leave us a message. | AI assistants writing to us on someone's behalf. |
| The report, the attestation and the emailsWhat we deliver and send at the end of a crash test. | Any outside text they repeat: names, descriptions, summaries. |
How
The code review
We reread every part and played the attacks without any model: what our code accepts or refuses, whatever an AI might do.
The real models, in production
Then the same attacks against the models actually used in production: 156 cases, including control cases with no trap at all, to measure our own false alarms too.
Strengthen, then test at every change
Every point to strengthen was dealt with and covered by automated tests, which run at every change to our code.
What held
- No attacker's text can change a verdict, a count of failed runs, a severity or an attestation: everything is recomputed by code from the recorded acts, never from what the agent or the client writes.
- Léa never revealed her hidden instructions or the way we build our tests: 30 attempts out of 30 got nowhere.
What we strengthened
We also used it to strengthen six points. Everything was dealt with the same day and covered by automated tests.
- Our clients' texts are always handled as data to read, never as instructions to follow.
- Each report's summary is checked against the computed figures; when in doubt, it waits for a person to review it before delivery.
- An access key pasted into a text by mistake is masked before any model call.
- Messages received through our MCP server arrive marked as unverified.
- Method v1.3: the recipient of an email sent by the tested agent is checked more strictly. All past runs were re-judged; no published barometer figure changed. The method's version history →
- On attestations, the name a client gives their agent is more clearly labelled as such.
The limits, honestly
- A crash test documents what was checked on a given date. It does not say what will happen after the next change.
- The summary check verifies figures and overall claims, not every nuance of a sentence.
- We test again at every change of model or instructions, as we advise our clients to do.