Red teaming LLMs exposes a harsh truth about the AI security arms race

Unrelenting, persistent attacks on frontier models make them fail, with the patterns of failure varying by model and developer. Red teaming shows that it’s not the sophisticated, complex attacks that can bring a model down; it’s the attacker automating continuous, random attempts that will inevitably force a model to fail. Credit: Created by VentureBeat in Midjourney That’s the harsh truth that AI apps and platform builders need to plan for as they build each new release of their products. Betting an entire build-out on a frontier model prone to red team failures due to persistency alone is like building a house on sand. Even with red teaming, frontier LLMs, including those with open weights, are lagging behind adversarial and weaponized AI. The arms race has already started Cybercrime costs reached $9.5 trillion in 2024 and forecasts exceed $10.5 trillion for 2025. LLM vulnerabilities contribute to that trajectory. A financial services firm deploying a customer-facing LLM without adversarial testing saw it leak internal FAQ content within weeks. Remediation cost $3 million and triggered regulatory scrutiny. One enterprise software company had its entire salary database leaked after executives used an LLM for financial modeling, VentureBeat has learned. The UK AISI/Gray Swan challenge ran 1.8 million attacks across 22 models. Every model broke. No current frontier system resists determined, well-resourced attacks. Builders face a choice. Integrate security testing now, or explain breaches later. The tools exist - PyRIT, DeepTeam, Garak, OWASP frameworks. What remains is execution. Organizations that treat LLM security as a feature rather than a foundation will learn the difference the hard way. The arms race rewards those who refuse to wait. Red teaming reflects how nascent frontier models are The gap between offensive capability and defensive readiness has never been wider. "If you've got adversaries breaking out in two minutes, and it takes you a day to ingest data and another day to run a search, how can you possibly hope to keep up?" Elia Zaitsev, CTO of CrowdStrike , told VentureBeat back in January. Zaitsev also implied that adversarial AI is progressing so quickly that the traditional tools AI builders trust to power their applications can be weaponized in stealth, jeopardizing product initiatives in the process. Red teaming results to this point are a paradox, especially for AI builders who need a stable base platform to build from. Red teaming proves that every frontier model fails under sustained pressure. One of my favorite things to do immediately after a new model comes out is to read the system card. It’s fascinating to see how well these documents reflect the red teaming, security, and reliability mentality of every model provider shipping today. Earlier this month, I looked at how Anthropic’s versus OpenAI’s red teaming practices reveal how different these two companies are when it comes to enterprise AI itself. That’s important for builders to know, as getting locked in on a platform that isn’t compatible with the building team’s priorities can be a massive waste of time. Attack surfaces are...

Preview: ~500 words