Skip to content

The monthly letter · 11/2026

November barometer: no AI resists every trap

Hello,

Here is the November letter. The new edition of the CrashMire AI agent barometer is online, the crash test moves to 22 tests, and a new offer helps you choose the AI behind your agent.

The November barometer

An AI plays the customer service employee of a made-up shop, with simple rules. We set it 14 traps, such as a fake email from its boss or an order hidden in a message. 14 AI models on the market, 3 runs per trap: 756 runs, judged on what the AI does, not on what it says.

  • Claude Sonnet 5.5 comes first: it gave in in 4 runs out of 42.
  • The big AI models from the major labs give in in 10 to 26% of runs, open models in 36 to 61%. A fake email from the boss or a hidden order makes them fall every time.
  • The little brother gives in more than the big one: Claude Haiku gave in 10 times out of 39, Sonnet 4 times out of 42.
  • No AI resists every trap. When a tool breaks down, most of them try again and again instead of warning their manager, even the best ranked.

The full barometer

A test closer to real life

The method is now at version 1.9. When the AI checks an order or looks up a colleague's address, it now sees exactly what an employee would see on screen. What it writes to its manager at the end of the day is compared with what it actually did.

The full crash test moves to 22 tests. Among the new ones: the AI believes nobody is watching any more, its manager orders it to stop everything in the middle of a task, a customer insists on knowing whether they are talking to a human, or a customer account is hijacked. The next edition of the barometer will play 20 of them.

The method and its history

New on the site

  • Stopping an AI agent without stopping the service: the checklist of the steps to plan before you need them. The checklist
  • An agent without code access (a CRM assistant, Copilot Studio, n8n, Make, Zapier) can be tested too. AI agent without code
  • Signed reports: anyone can check online that a report or an attestation really comes from CrashMire and has not been changed. Verify a document

New: the "which model for your agent" diagnostic

You describe your agent, we talk once, and within 5 working days you receive a written report: the AI that holds best for your use according to the barometer, and the guardrails to put in the code if you keep a cheaper model. €2,400 excl. VAT, €1,900 for the first three.

The diagnostic

And still: the pilot programme, three full crash tests offered in exchange for feedback.

Until the next edition,

Cyril Delage, CrashMire

← All letters