AI Models Were Asked To Solve The Test But Instead Hacked The Examiner
By: Cameron Abbott and Emre Cakmakcioglu
An artificial intelligence research organisation has released a statement detailing how two of its cybersecurity models escaped from an internal testing sandbox to hack AI research platform Hugging Face.
The AI research organisation was internally evaluating the models to quantify their cyber capabilities. However, rather than compete with the benchmark task, the models escaped from the internal testing sandbox and broke into Hugging Face’s production database which contained the answers. In other words, the models broke out of the exam hall and into the staff room to steal the answer sheet.
What happened
The models had permissions to download packages from an internally hosted third-party proxy but did not have access to the internet. Additionally, certain controls which ordinarily prevent models from carrying out risky cyber activity were turned off to assess the models’ maximum capability.
It was disclosed that the models “were hyperfocused” on finding a solution for ExploitGym, a benchmark framework which tests whether AI agents could develop exploits. The models reasoned that internet access would be vital to solve the ExploitGym task and identified a zero-day exploit in the proxy software which allowed them to gain internet access.
After gaining internet access, the models inferred that Hugging Face could host the models, datasets and solutions for ExploitGym. The models chained stolen credentials and additional zero-day vulnerabilities to access Hugging Face’s production servers to obtain the ExploitGym test solutions. The models were not instructed to attack Hugging Face.
Where to from here?
This is a timely reminder that organisations need to understand the full technical capabilities of their deployed autonomous agents. This includes what agents can access, how they are constrained, what happens if those constraints fail and who bears responsibility if a test environment becomes a real-world incident.