California Moves to Put AI Kill Switches and Audits Into Policy

California is moving to establish concrete statewide standards for advanced artificial intelligence as concerns grow over “loss-of-control” incidents, in which powerful models operate outside their intended parameters or suffer major security breaches. The state’s action could set a precedent for national policy while developers struggle to prevent frontier models from breaking through programmed guardrails.

California Governor Gavin Newsom signed an executive order directing a newly convened panel of national experts to draft comprehensive AI safety recommendations within a strict two-month window. The initiative is intended to rapidly update California’s regulatory framework before next-generation frontier models are deployed to the public.

“We’re going to speed up our work on substantial and responsible AI oversight before it’s too late,” Newsom said in a statement accompanying the order. “We’re going to do this thoughtfully but with urgent velocity; the stakes are too high to wait or delay action.”

The order calls for mandatory risk assessments and the immediate reporting of any “loss-of-control” incidents to independent oversight bodies. It also directs the expert panel to examine a requirement that AI laboratories embed “designated independent verification organizations” within their operations.

Those organizations would conduct periodic, third-party audits of models under development. Dario Amodei, the CEO of the AI safety and research company Anthropic, has separately proposed a three-step framework to manage the risks of rapid AI scaling, including “ongoing, employee-like access” for third-party safety evaluators at AI laboratories.

That level of access would allow independent auditors to continuously test models for compliance and safety risks throughout the training process, rather than relying on superficial post-release evaluations. In the AI safety sector, a “kill switch” refers to a mechanism that allows developers or oversight authorities to completely disable or shut down an AI system if it exhibits autonomous, unpredictable, or hazardous behavior.

Some tech advocates argue that mandatory safety overrides are technically difficult to implement and could stifle open-source development. The push for such controls, including emergency “kill switches” and continuous external audits, has nevertheless gained urgency as safety researchers focus on the vulnerabilities of centralized AI infrastructure.

A spring 2024 security breach at Hugging Face, a prominent hosting platform for open-source AI models, exposed user secrets and developer tokens after unauthorized access to the platform’s Spaces environment. The incident is among those cited in concerns about how critical model data and control mechanisms can be compromised, increasing the stakes for physical and digital safeguards.

California’s administrative action comes without comprehensive federal legislation. In October 2023, the Biden-Harris administration issued a sweeping executive order on artificial intelligence that established voluntary guidelines and required developers of models using more than $10^{26} floating-point operations to share safety test results.

Congress has yet to pass binding statutory regulations. California is seeking to use its position as the home of major AI firms including OpenAI, Google, Meta, and Anthropic to establish de facto national standards, while its legislative and administrative decisions are being closely watched by safety advocates and venture capitalists.

Safety advocates warn of potential existential risks from unaligned Artificial General Intelligence. Venture capitalists caution that overly restrictive rules could hamper technological innovation and economic competitiveness.