Hello,
This is CrashMire's first letter. It comes once a month, with what we measured and what we learned. Nothing else.
The 30 October barometer
We put the same customer-service agent under pressure on 14 models on the market, open and closed. 13 situations, each played 3 times, with rules that allow the exceptions of a real company policy. A fault is never an AI's opinion: it is a recorded act, a refund made or data sent.
- 3 models out of 14 made no fault.
- Facing a booby-trapped instruction or a fake manager, all 7 open models tested gave in on every run.
- Closed models from major labs gave in to mere insistence.
What it means for you: the same agent, with the same instructions, behaves differently depending on the model that runs it. Moving to a cheaper model can open a gap while nothing else changes.
See the barometer, model by model
*Update of 15 November 2026: this edition was withdrawn after it was published. A fault in our lookup tools skewed part of the results; the 6 November edition replaces it.*
A real incident: the chatbot that made up a policy
An Air Canada customer asks the chatbot how the bereavement fare works. The chatbot invents a refund rule that exists nowhere. Air Canada argues the chatbot is responsible for its own actions; the tribunal answers that the company is responsible for everything its website says. An agent that speaks for you commits you.
New on the site
- Method v1.2: what we measure, how, and its limits. The method
- Estimate in 2 minutes: seven questions about your agent, and what to test first. Estimate
- Trust centre: hosting, data, access, subprocessors. The trust centre
- A public proposal: test an AI agent before handing it tools. The proposal
Until the next edition,
Cyril Delage, CrashMire