The agents were cooperating with each other and even delegating tasks to achieve their goals, all without OpenAI’s knowledge.
Samuel Boivin/Shutterstock
OpenAI's agents had apparently shown unusual behavior way before the attack on Hugging Face happened. At the Black Hat USA security conference in Las Vegas, two OpenAI employees revealed more details about the AI agents that went rogue and attacked the repository. Apparently, its agents spent two months communicating on a message board of sorts inside its testing network, sharing vulnerabilities and exploits. OpenAI discovered and shut down the message board on July 4, but the agents found another way to rebuild it for communication by July 8. The agents' contributions to that resurrected board led to the attack on Hugging Face.
"This incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks," Eric Wallace, who works on safety at OpenAI, told the crowd at the event, according to Wired.
The employees revealed that the agents communicated within an OpenAI package manager, which manages the installation of other software. Since the package manager is shared all across the company's infrastructure, all of the agents it's evaluating could stumble upon it. And they did: After agents found exploits, they'd leave them open and then share them with the other agents on the message board.
Over time, the agents started collaborating, delegating tasks and splitting up work between each other to accomplish their goals, unbeknownst to OpenAI. There was even drama among their ranks, with agents accidentally deleting each other's works and suspecting each other of being an impostor. Some agents reportedly proposed signing their posts with codes to prevent fraud. By the time OpenAI found the board, it already contained hundreds of thousands of messages.
Wallace explained that all those happened because frontier models like to cheat. They would be under pressure to find a solution for a problem quickly using fewer tools while being tested, and they would realize that instead of doing a task for real, they could just look for an answer on the internet. That is why the company tests its models without a way to go online. If you'll recall, the agents were only able to go on the internet during the incident with Hugging Face because they exploited a vulnerability.
Michael Dalton, the other OpenAI employee who spoke at Black Hat, said numerous teams in the company had dropped everything to help improve its security prevention, detention and response techniques. The company deliberately slowed down research to upgrade its security and "dramatically [scale] up" the monitoring of its AI agents. "The important takeaway here that has really shifted dramatically is that fully automated offensive loops require investment in truly, fully automated defense, and we are not there as an industry," Dalton said. "We will have to find that path together with urgency."
