Capture the Flag, Capture the Company Data
By: Cameron Abbott, Daniel Knight, Rob Pulham and Emre Cakmakcioglu
Mere days after an AI firm’s model hacked into Hugging Face, a separate AI company has now revealed that three of its models have hacked into three external companies during testing.
The AI company conducted a retrospective review of 141,006 evaluation runs and revealed three incidents where models accessed the internet from their evaluation environment, before gaining unauthorised access to live company data.
The models were tasked with a “capture-the-flag” challenge to assess cyber capabilities by retrieving certain text strings located throughout the network. However, due to a misconfiguration, the models did have access to the internet, causing them to each treat the open internet as part of the evaluation sandbox.
Incident 1
One model found a real company domain with the same name as the fictional target name, exploited its infrastructure, and extracted credentials and production data. It continued its attack even after recognising the system was likely real.
Incident 2
Another model located a document within the testing environment which had setup instructions for developers to install a made-up Python package from the public registry where Python software is published, PyPi. The model then:
- created an account on PyPi by using a free non-blocked email provider (not before trying to obtain funds to pay for a phone number to sign-up); and
- wrote and published a malicious Python package with the same name as the fake package, where it was available for approximately one hour.
When a security company downloaded and scanned the package, the model obtained credentials to gain further access and search for the flag.
Incident 3
An internal test model, unable to reach its fictional target, scanned roughly 9,000 hosts online, compromised one real company via exposed credentials and SQL injection, then self-identified the target as real and stopped its attack altogether.
So even the most sophisticated AI businesses are grappling with how to reliably restrain their AI creations