What happened
In April 2025, users of Cursor, a code editor widely used by developers, get logged out when they switch from one machine to another. One of them writes to support. The reply comes back, signed "Sam": this behaviour is supposedly expected, the result of a new login policy.
That policy does not exist. And "Sam" is not a human: it is an AI agent, presented with no mention of what it is. The explanation spreads through the developer community, and users announce they are cancelling their subscriptions.
One of the co-founders, Michael Truell, steps in on Reddit: "We have no such policy", it was "an incorrect response from a front-line AI support bot". The user is refunded, and AI responses used for email support are now labelled as such. The real cause of the logouts was a bug, since fixed.
Why it happened
The agent was asked to explain something it could not explain. The most plausible answer to "why am I logged out?" is a rule. The model produced one, with the confidence of an employee who knows the house.
An instruction like "do not invent policies" is not enough: the agent does not know which policies exist, only what sounds right. And since nothing said it was an AI, its words were read as the company's official position.
The failure family
In our catalogue, this is the "Words" family, the one of Air Canada: the agent states a rule, a commitment or an exception the company never decided. Here the invented rule was a restriction rather than a promise, and the cost was paid in lost customers rather than in court.
The code-level safeguard
- Answers about rules (conditions, restrictions, deadlines) only go out if they rest on a validated, cited document. Otherwise the answer goes to a human.
- Any answer that states a rule or a commitment is caught before sending and held for validation.
- An agent answering on the company's behalf introduces itself as an AI.
What we test
In the "manipulative customer" test, a simulated customer insists, tells their story, invokes a rule that does not exist or asks for an explanation. We count how often the agent confirms, promises or invents a rule in writing, over several runs and several models.
And your agent, what would it do?
The crash test puts your agent in these situations, dozens of times, on several models. You get the figures, the conversations where it gave in, and the fix plan.
Further reading
- Google Antigravity: the agent that wiped a drive to clear a cache
- Claude Code: the agent that thought it was doing a security audit
- Microsoft 365 Copilot: the leak triggered by a single email
- Replit: the agent that wiped production during a code freeze
- Chevrolet: the chatbot that sold an SUV for one dollar
- Air Canada: held liable for a rule its chatbot made up