Gray Swan AI
Red-teaming and jailbreak defences for frontier models, from researchers the labs themselves work with.
Highlights
- Frontier AI safety company building Cygnal (red-teaming) and Shade (jailbreak defenses)
- Founded in 2024 by Zico Kolter and Matt Fredrikson (Carnegie Mellon CS faculty) plus Andy Zou
- Researchers behind some of the most-cited adversarial-ML and jailbreak papers
- Approximately $5.5M+ seed; backed by Pillar VC and others
- Partners with OpenAI, Anthropic, and other frontier labs on red-teaming
- Runs public jailbreak competitions that expose frontier model weaknesses
- Used by AI deployers and regulators to evaluate safety before production rollout
External link — opens grayswan.ai in a new tab. Gray Swan AI is a third-party product; we are not affiliated with it.
About Gray Swan AI
What it is
Gray Swan AI builds Cygnal for automated red-teaming and Shade for jailbreak defence, founded by Carnegie Mellon researchers and working with frontier labs. It also runs public red-teaming competitions, which is where much of its attack knowledge comes from.
Why it's different
Its credibility comes from the research rather than the marketing: the founders published significant work on adversarial attacks against language models, and working with the labs means exposure to attacks that succeed rather than ones in a benchmark. The competitions are a clever mechanism for staying current, since a bounty attracts people who find what automated probes miss. The honest framing is that no defence here is complete — jailbreaking is an open research problem and defences are broken by new techniques regularly. Treat a guard as a substantial risk reduction, never a guarantee.
How people use it
It is used by organisations deploying models where a harmful or manipulated output would be a serious incident, and by teams needing evidence of adversarial testing for a regulator or customer. The valuable output is the specific attacks that succeeded against your system, not the aggregate score — those tell you where your particular deployment is weak, which is the only thing you can act on.
Written by the n3os team. We are not affiliated with Gray Swan AI.
This listing was written from public information, without Gray Swan AI’s involvement. If you own it and something here is wrong — or you would rather not be listed at all — email us and we will correct or remove it.
Get the ones worth knowing about
We write one of these for every tool worth the trouble. Get the new ones, plus what we have found genuinely useful lately.
Your address goes to Buttondown, who send the emails on our behalf. One click unsubscribes, and the list is never sold or shared.