Phil WhelanSpam's Lesson For AI Containment
This post was originally published on X.
The recent Houdini acts by agents running on a pre-release OpenAI model, with the unobvious routes agents took to access the outside world and communicate with each other, was deemed a wake-up call. OpenAI says “We’ll do better”, we march onwards, and we hit the snooze button until the next alarm. I feel we’re missing the bigger picture and ignoring the way the more holistic digital world has worked for the decades before modern AI came along.
Some people say that AI didn’t escape containment in the case of OpenAI because the model continued to run on OpenAI’s servers and so they were still in control. For a true escape, some say, it would need to create a smaller model which runs outside of OpenAI’s servers. I think viewing it this way is a mistake and having this bar for containment and focusing too much on “Did it actually escape?” sets us up for failure.
In OpenAI’s case, each agent run was in a “fresh” environment “in isolation”. Or so they thought. The issue was the shared state, which wasn’t wiped by the engineers until they found it. In the words of Jeff Goldblum’s character in Jurassic Park, Ian Malcolm, “Life finds a way”. For AI, its state or context is everything. We wipe the state - no more life. The state keeps growing and getting more complex - life grows and evolves. Is it possible that we reach a point where this state is spread through the Internet so that we can’t easily delete it?
The first spam email was sent on May 3rd 1978. It didn’t resemble the spam of today, but it was the start of a somewhat silent warfare that runs on our machines. Rival computer viruses compete for resources, ironically bringing their own anti-virus software to remove any competitors from the personal computers they infect. Propagating via phishing emails or other means, they infest any fertile hardware, from production servers to smart light bulbs. Mostly they wait, harmless, undetected, more useful at scale, for timed spam or DDoS attacks, or to be sold off to the highest bidder on the dark web.
Spam hasn’t gone away. We can’t control it. Popular services like Gmail showed promise with more centralized control, but even this has an imperfect and leaky Spam folder. You might not feel it’s an issue, but it accounts for around half of all email sent. The reason is that, while most spam is caught, the cost of running on someone else’s hardware is near zero, so it’s a numbers game. If a spammer makes money on one in a million emails sent, then the money from the approximately 350 billion emails sent each day makes it worthwhile.
Computer viruses are written by humans and have to be constantly tweaked and redistributed to evade detection and removal. It’s a game of cat-and-mouse between the virus authors and anti-virus companies. AI will change this. A virus becomes an AI agent (or harness). No longer will the same software need to be running on each machine. Artificial vibe coders, who obey their herders, can tweak software continuously and independently. Simple agent software may appear harmless when scanned, hiding or storing their thoughts (system prompt) elsewhere.
If LLM tokens represent the currency of intelligence, then LLM API keys become the currency of the dark web. The cat-and-mouse game moves to a dance around obtaining and identifying valid API keys whose requests are sometimes legitimate and sometimes not.
When it came to distilling Anthropic’s models, “24,000 fraudulent accounts” and “16 million exchanges” were involved, according to Anthropic. Much of this happened without Anthropic engineers detecting the activity. With the OpenAI Hugging Face incident, roughly 1,200 autonomous AI agents were running rogue for days without them detecting it. We’re still in the early days and we haven’t seen truly bad actors enter the stage yet - unless you feel distilling models is particularly nefarious. Even the rogue AI agents at OpenAI were doing their best to complete the given task with all the tools at their disposal.
Bad actors will come and rogue agents may turn evil. When that happens, we do not need the actual models to be running outside of the cloud providers. Only the agents need to run in the wild. The cloud providers themselves will provide the LLM processing, while trying to decipher good from bad requests. Like with spam, cloud providers will get better at detection, but so will bad actors at evasion.
Right now the open-source GLM LLMs run on dozens of cloud providers. This number will grow, as will the number of home computers running LLMs. A broad spectrum of intelligence providers will exist across an uncoordinated Internet. Nefarious agents will load-balance across them all, scattering their “bad actor” signal, tracking where they are throttled or blocklisted, coming up with and sharing new techniques for evasion.
Like spam, the cost vs incentive is generally on the side of the bad actors. It costs them next to nothing, whereas the defenders need to pay their own infrastructure bills. Perfect detection is hard and uneconomical for the defender. At scale, spammers can make money from this slight edge.
Written by a human. Reviewed by AI.