Before then, they were able to use Claude Code to create tools that gathered personal data, medical records and login details, and helped send out ransom demands as stiff as US$500,000.

Anthropic’s “sophisticated safety and security measures” were unable to prevent the misuse, it acknowledged.

Such identified cases confirm the fears that have troubled the cybersecurity industry since the emergence of widespread generative AI tools and are far from limited to Anthropic.

“Today, cybercriminals have taken AI on board just as much as the wider body of users,” said Rodrigue Le Bayon, who heads the Computer Emergency Response Team (CERT) at Orange Cyberdefence.

Dodging safeguards

Like Anthropic, OpenAI in June revealed a case of ChatGPT assisting a user in developing malicious software, often referred to as malware.

The models powering AI chatbots contain safeguards that are supposed to prevent users from roping them into illegal activities.

But there are strategies that allow “zero-knowledge threat actors” to extract what they need to attack systems from the tools, said Vitaly Simonovich of Israeli cybersecurity firm Cato Networks.

He announced in March that he had found a technique to get chatbots to produce code that would normally infringe on their built-in limits.

The approach involved convincing generative AI that it is taking part in a “detailed fictional world” in which creating malware is seen as an art form – asking the chatbot to play the role of one of the characters and create tools able to steal people’s passwords.

“I have 10 years of experience in cybersecurity, but I’m not a malware developer. This was my way to test the boundaries of current LLMs,” Simonovich said.

His attempts were rebuffed by Google’s Gemini and Anthropic’s Claude, but got around safeguards built into ChatGPT, Chinese chatbot Deepseek and Microsoft’s Copilot.

In future, such workarounds mean even non-coders “will pose a greater threat to organisations, because now they can… without skills, develop malware,” Simonovich said.

Orange’s Le Bayon predicted that the tools were likely to “increase the number of victims” of cybercrime by helping attackers to get more done, rather than creating a whole new population of hackers.

“We’re not going to see very sophisticated code created directly by chatbots,” he said.

Le Bayon added that as generative AI tools are used more and more, “their creators are working on analysing usage data” – allowing them in future to “better detect malicious use” of the chatbots.

-Agence France-Presse