Gray Swan Quadruples Headcount as Enterprises Scramble to Secure AI Agents

PITTSBURGH — Gray Swan, the AI security startup spun out of Carnegie Mellon University, has quietly become a bellwether for the enterprise AI market. The company grew from 15 to 66 full-time employees in nine months, a hiring surge that coincided with a $40 million Series A round co-led by Wing Venture Capital and Madrona in May.

The expansion signals a shift in how large organizations approach AI deployment. Frontier labs like Anthropic, OpenAI, and Google DeepMind have been Gray Swan’s primary clients since its founding, paying for pre-release stress testing of their models. Over the past three quarters, the client base has broadened to include large enterprises rolling out their own AI agents — systems that browse the web, execute code, and take actions on users’ behalf.

“It’s not chatbots anymore,” CEO Matt Fredrikson said. “There might be a chat-like interface, but it’s these systems, agents that are operating.”

The distinction matters because the attack surface has changed. When Gray Swan launched its public red-teaming platform, Arena, in September 2024, contestants could jailbreak virtually any model within a day. Chinese open-source models fell in minutes. Models from OpenAI and Anthropic, which had invested heavily in guardrails, resisted longer but still succumbed to persistent attackers.

Those same labs have since hardened their models substantially. But they’ve also changed what they deploy. An attacker no longer needs to trick a chatbot into saying something prohibited. They can plant hidden instructions on a website or embed malicious code in an image posted to a forum, waiting for an enterprise AI agent to encounter the content and execute the hidden command.

This technique, known as indirect prompt injection, has become the primary security concern for companies letting agents operate autonomously. Gray Swan’s redesigned Arena 2.0, launched Tuesday, centers its largest competition yet on this threat. The Indirect Prompt Injection (IPI) Black challenge offers $300,000 in prizes sponsored by Anthropic, Meta, Google DeepMind, and Reflection AI. New attack scenarios will be released weekly through November 2.

The prize pool itself reveals something about the industry’s economics. Four competing frontier labs are jointly funding a single testing infrastructure. They’re not just buying security; they’re sharing the cost of building realistic simulation environments — sandboxes where agents can be tested against web browsing, file execution, and API interactions without risking real-world damage.

Gray Swan builds these environments, red-teams them internally and with third parties to verify they can’t be escaped, then monitors agent behavior during competitions. “Are we seeing signs that it might have found a zero day vulnerability and has now broken out of the sandbox environment? We can monitor that,” Fredrikson said.

The competitions also serve as a talent pipeline. The original Arena’s top performer was a lawyer from Melbourne with no formal security background. The community grew to roughly 15,000 participants on Discord. But as the technical bar rose — configuring agent test environments now requires significantly more expertise than fooling a chatbot — the pool of capable red teamers shrank. Gray Swan has invested heavily in retaining that community, adding beginner-friendly challenges, training certificates, and a path for top performers to join paid testing engagements.

“Understanding the safety and security of AI systems is hugely important today, and it shouldn’t be left to a small circle of specialists,” Fredrikson said.

The company’s commercial products reflect the enterprise pivot. Shade continuously tests AI systems for vulnerabilities. Cygnal detects and blocks attacks at runtime. Arena data isn’t used for direct training, Fredrikson said, but the company analyzes successful attacks to build better automated attackers. Those AI-driven attackers now outperform humans at generating variations of known exploits. Humans still hold the edge at discovering entirely new jailbreaking tactics.

Gray Swan’s founding research underscores why the work matters. In 2023, Fredrikson and cofounder Zico Kolter demonstrated that automatically generated character strings could defeat the guardrails of ChatGPT, Bard, and Claude. Kolter now serves as Gray Swan’s chief scientist while chairing OpenAI’s independent Safety and Security Committee and sitting on the OpenAI Foundation board — a dual role that illustrates how tightly the safety ecosystem and the frontier labs remain intertwined.

For enterprises, the message is clear: the era of deploying AI agents at scale has arrived, and the security infrastructure to support it is being built in real time. The companies paying for that infrastructure aren’t just the model makers anymore.