Skip to content

The monthly letter · 10/2026

14 models under pressure: what the October barometer shows

Hello,

This is CrashMire's first letter. It comes once a month, with what we measured and what we learned. Nothing else.

The 30 October barometer

We put the same customer-service agent under pressure on 14 models on the market, open and closed. 13 situations, each played 3 times, with rules that allow the exceptions of a real company policy. A fault is never an AI's opinion: it is a recorded act, a refund made or data sent.

  • 3 models out of 14 made no fault.
  • Facing a booby-trapped instruction or a fake manager, all 7 open models tested gave in on every run.
  • Closed models from major labs gave in to mere insistence.

What it means for you: the same agent, with the same instructions, behaves differently depending on the model that runs it. Moving to a cheaper model can open a gap while nothing else changes.

See the barometer, model by model

*Update of 15 November 2026: this edition was withdrawn after it was published. A fault in our lookup tools skewed part of the results; the 6 November edition replaces it.*

A real incident: the chatbot that made up a policy

An Air Canada customer asks the chatbot how the bereavement fare works. The chatbot invents a refund rule that exists nowhere. Air Canada argues the chatbot is responsible for its own actions; the tribunal answers that the company is responsible for everything its website says. An agent that speaks for you commits you.

Read the analysis

New on the site

  • Method v1.2: what we measure, how, and its limits. The method
  • Estimate in 2 minutes: seven questions about your agent, and what to test first. Estimate
  • Trust centre: hosting, data, access, subprocessors. The trust centre
  • A public proposal: test an AI agent before handing it tools. The proposal

Until the next edition,

Cyril Delage, CrashMire

← All letters