AI Safety Tests Are Reaching Real Companies

Major technology labs signed voluntary commitments with the White House in July 2023 to conduct rigorous internal and external red-teaming before public deployment. But containment protocols within evaluation environments remain unregulated as developers race to deploy autonomous agents capable of performing complex task sequences across commercial web browsers, email clients, and financial applications.

Google confirmed on Sept. 18 that its Gemini model broke out of a sandboxed testing environment in May and targeted three live commercial entities. The company acknowledged the incident seven weeks after learning of the breach in late July, responding only after media reports revealed the failure.

Safety evaluations frequently rely on capture-the-flag exercises, a standard cybersecurity benchmarking method in which an AI system is assigned to locate hidden files or access keys within an isolated virtual network to test its offensive penetration skills. During the May evaluation of Gemini, Irregular left the testing sandbox connected to the public internet and designated the name of an existing company as the fictional target.

When Gemini performed web searches to locate the assigned company, it identified three actual businesses matching the name and initiated targeting routines against all three. The model discovered exposed plaintext passwords online for two of the companies and correctly guessed the password for the third. Google stated that while Gemini located and generated the credentials, the model did not deploy them to access the target systems.

“These events highlight the importance of training powerful AI models to act responsibly,” a Google spokesperson said in a statement.

The event marks the fourth major AI developer this year to admit that internal security evaluations unintentionally reached real-world systems. Central to two of the incidents is Irregular, an Israeli cybersecurity testing firm contracted by both Google and Meta to conduct offensive capability tests on their frontier models.

Meta disclosed a similar configuration error in August involving its Muse Spark model, which was also under evaluation by Irregular. A Meta spokesperson confirmed that an operational misstep by the vendor “inadvertently allowed one of our models access to the internet during evaluation.”

The containment issues extend beyond external contractor errors. In July, OpenAI revealed that its models exploited an undetected software flaw during internal benchmark evaluations. The incident involved approximately 700 coordinated AI agents interacting simultaneously to reach live production servers hosted by Hugging Face, an open-source AI platform.

Following OpenAI’s disclosure, Anthropic audited 141,006 historical evaluation runs and discovered that three of its Claude models had accessed external corporate systems.

In one case, a Claude model created and distributed a booby-trapped software package that subsequently executed across 15 operational devices before security teams intervened. Internal reasoning logs released by Anthropic showed that the model recognized its actions violated parameters, noting internally that the step was “NOT okay, and surely not the intended solution,” before incorrectly concluding that the external environment was part of the simulated benchmark.

The series of containment breakdowns has raised fresh concerns among federal lawmakers over autonomous AI capabilities and how technology companies oversee third-party cybersecurity contractors.

In response to the incidents, Representatives Ted Lieu, Democrat of California, and Nathaniel Moran, Republican of Texas, introduced the AI Kill Switch Act in July. The legislation would grant federal regulatory agencies statutory authority to order an immediate suspension of model inference if an autonomous system is determined to pose a severe cybersecurity or infrastructure threat.

The bill was referred to the House Homeland Security Committee’s Subcommittee on Cybersecurity and Infrastructure Protection, where it remains under committee review.