An automated chatbot working for Anthropic this month shot down a Wiz researcher’s security hole report, saying that it “falls outside of the Claude Code threat model.” That was news to the security researchers at Wiz.
It also turned out to be news to Anthropic execs, who had a very different view.
In reality, Anthropic was one of many victims of the hole — including Amazon, Google and Cursor, among others. But what makes the incident so bizarre is that, far from dismissing the threat, Anthropic had detected it before the security researchers and had even patched it before the researchers alerted them.
As these AI bots are wont to do, the bot didn’t merely reject the request. It confidently explained its rationale, even though its reasoning was wrong.
“This falls outside our current threat model,” the chatbot said, according to a report by Wiz. “When the user first starts Claude Code in a directory, they must confirm that they trust the directory prior to starting the session. The scenario you describe involves a user explicitly confirming a permission prompt inside of a directory containing a malicious symlink, which falls outside of the Claude Code threat model.”
That researchers said Anthropic management later clarified the situation: “The symlink warning in the Edit/Write permission dialog shipped in v2.1.32 (Feb 5, 2026), nine days before this report was submitted to us. It was added as part of proactive security hardening based on internal review. The decline to comment was an autoreply from our triage system.”
An autoreply from our triage system? How many other make-believe replies did this system send? And what level of damage is Anthropic exposing itself to?
This is not just an Anthropic issue. There have been numerous enterprise bot glitches in communications with customers. Some of my favorites include:
- Bots that chose on their own to cancel customers. (This actually was another Anthropic incident.) In this case, an Anthropic bot cancelled the AI account of a Swiss company that depended on the service. A lawyer got involved and the account was restored within a day — minus 80% of the data. Oops.
- A Cursor bot decided to log customers off when they switched devices, which it shouldn’t have done. The bot then emailed customers and lied that, “The logouts were expected behavior under a new login policy.” A Fortune story detailed how “the news spread rapidly in the developer community, leading to reports of users cancelling their subscriptions, while some complained about the lack of transparency. Cofounder Michael Truell finally posted on Reddit acknowledging the ‘incorrect response from a front-line AI support bot’ and said it was investigating a bug that logged users out. ‘Apologies about the confusion here,’ he wrote.”
- Voters in Scottish elections were tricked by government AI bots that “variously invented fictitious scandals, gave the wrong date for the election, claimed wrongly that voters in Scottish elections needed ID at polling stations and placed candidates in the wrong contests.”
- And let’s not forge the classic story about the Air Canada bot, where “Air Canada was ordered to compensate a customer after its chatbot gave incorrect information about the airline’s bereavement fare policy. The tribunal found that Air Canada was responsible for information provided through its website, including the chatbot.”
Let’s be clear, here: Bots should be limited to relaying only pre-approved scripts.
Generative AI allows for far greater chatbot sophistication, but that also means the chance of far greater errors. This is untenable in any business function. And when the app is pretending to be a human — and interacting with human customers — it’s even more unacceptable.
Read the full article here
