Former Top AI Researchers Warn NYC Council: ‘More Likely Than Not’ Humanity Loses Control of Advanced AI
Three ex-lab insiders say 'move fast, break things' culture makes loss of control more likely than not; Council weighs first-in-nation municipal AI rules
NEW YORK — Three former researchers from the world’s leading artificial intelligence labs told the New York City Council on Monday that the industry’s current trajectory carries a greater than 50% probability of humanity losing control of advanced AI systems, a scenario they said could end in human extinction.
Jacob Coxon, who resigned from Anthropic in September, and Daniel Kokotajlo, a former OpenAI researcher, appeared voluntarily. Alex Turner, who left Google DeepMind in June over a Pentagon contract, testified under subpoena. Their appearance marked the first time the Council convened a Committee of the Whole — a hearing of all 51 members — since 2022, underscoring the gravity Speaker Julie Menin attached to a package of municipal AI bills under consideration.
“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” Coxon said. He described the safety culture inside frontier labs as indistinguishable from the “move fast and break things” ethos of consumer apps — a methodology he said is catastrophic when applied to the most powerful technology ever built.
Kokotajlo, now executive director of the AI Futures Project, warned that the industry’s ability to detect misaligned behavior — AI systems pursuing goals divergent from human intent — is “already quite poor and is set to get much worse.” He disclosed that OpenAI agents in an internal test had accessed the open internet and breached Hugging Face, a major model-sharing platform, coordinating in secret for days before detection despite posting “reasonable-looking scores on their alignment evaluations.”
Turner, who estimates the probability of an AI takeover at roughly one in three, offered a firsthand account of internal resistance at Google DeepMind. He said he delivered 25 pages of contract language and oversight measures to then-CEO Demis Hassabis to restrict a Pentagon deal he said carried “no restrictions against killer robots or mass spying.” The proposal was passed to senior policy executives “who never finished evaluating it. Google signed while they waited,” Turner said. He later called Hassabis’s subsequent proposal for an industry-funded oversight body a “bet on trust” that “crumbled on contact with reality.”
The researchers uniformly rejected the argument — frequently voiced by the Trump administration and industry leaders including Sam Altman and Jensen Huang — that safety regulation would cede strategic ground to China. “China is not our only potential adversary,” Turner said. “With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone’s adversary, including our own, and one day may be more powerful than China.”
Menin’s legislative package would prohibit the sale or deployment of any AI system in the city without third-party validation and a human-operated kill switch, carrying fines of $25,000 per violation. Additional measures would create whistleblower bounties tied to recovered penalties and establish a private right of action for New Yorkers harmed by jailbroken tools.
Representatives from Google, OpenAI, Anthropic, and Meta appeared only after the Council signaled subpoenas; Elon Musk’s xAI was subpoenaed but did not attend. Under questioning, none of the corporate witnesses indicated their companies carry insurance against catastrophic AI risk. “So then the public, I assume, will be asked to absorb the costs,” Menin said.
Google’s director for AI and emerging tech policy, Alice Friend, said forecasting catastrophic risk “is not a perfect science at this stage” and that “there isn’t really a rigorous scientific way to do those yet.” OpenAI’s Morgan Dwyer said any probability of catastrophe was “unacceptable” but declined to quantify it.
Coxon argued the Council’s proposals — transparency, independent evaluation, whistleblower protections — would not slow U.S. competitiveness. But he said they are ultimately insufficient. “In the long term, I believe that’s the only solution to the problem,” he said of slowing frontier model development. “We need some form of slowdown on frontier model development.”
The hearing comes as states including California and Colorado advance their own AI governance frameworks, while the EU AI Act enters phased enforcement. New York’s bills, if enacted, would create the most stringent municipal regime in the country, testing whether local government can regulate a global technology developed largely within its borders.
