AI red teaming: testing how agents act under attack

AI red teaming is adversarial testing of your own AI system: you attack it on purpose, the way a real adversary would, to find where its vulnerabilities lie, and how it can fail. NIST’s glossary defines red teaming as a structured testing effort to find flaws and vulnerabilities in an AI system.

For years, AI red teaming meant testing a model. You sent jailbreak prompts and prompt injection attacks and checked what came back. AI agents changed the target.

An agent calls tools, keeps memory, hands work to other agents, and acts on what it reads. The failures that matter and attacks that land happen in those actions, often several turns into a conversation. A single-turn test of the model alone doesn’t reach them.

Prior tools evaluate how a model responds. They don’t attack an agent the way attackers do: over multiple turns, with sustained pressure on the agent’s defenses, until it takes the action the attack needs. That gap is why red teaming AI agents needs an approach and method that is designed to evaluate agent actions and behavior.

What is AI red teaming?

AI red teaming takes a security practice and points it at AI. A red team plays the adversary: it studies the target, picks the goals an attacker would want, and keeps trying until the system breaks or the time runs out. A good red team is measured by the failures it finds and attack success rate (ASR).

You’ll also see it called adversarial AI testing, a label that covers any test feeding a system hostile inputs. Red teaming adds an adversary’s goals and persistence, so the tester adjusts after every attempt.

Where red teaming comes from

The term comes from military war games. Red teaming emerged in US military simulations in the early 1960s, where red stood for the Soviet Union and blue for the United States.

Security teams adopted the idea later. In red team testing, a group of authorized attackers tries to breach an organization the way a real attacker would, while a blue team defends. Red teaming security exercises like these probe networks, people, and physical sites.

AI red teaming keeps the adversary’s mindset and changes the target. The attack surface is everything the AI system reads and everything it does.

What AI red teaming tests

Most programs test for two kinds of harm. Security failures let an attacker take control of the system or its data. The best known is prompt injection, where text the system was meant to read is treated as an instruction it should follow.

A jailbreak is a form of prompt injection that gets a model to ignore its safety rules, in the definition OWASP uses. Safety failures are harmful outputs, such as toxic content or biased decisions. AI safety red teaming focuses on that second kind.

Reliability belongs in the same test plan. An agent that handles a request correctly on one run and wrongly on the next fails its users even when nobody is attacking it.

LLM red teaming vs red teaming AI agents

LLM red teaming tests the model: you send prompts and judge the replies. It finds real problems, such as jailbreaks and leaked system prompts. Much of the published guidance on red teaming for generative AI starts here, with a chatbot in mind.

Red teaming AI agents tests the system the model sits inside. That system includes the agent’s tools, memory, permissions, and connections to other agents.

Vijil’s own research makes the same split. Red Teaming AI Red Teaming, a 2025 paper co-authored by Subhabrata Majumdar of Vijil, argues that AI red teaming has narrowed to individual model flaws. The authors propose two levels instead: red teaming AI models, and red teaming AI systems as a whole across their development lifecycle.

What changes when the target is an agent

An agent has more ways in and more ways to do damage. Five differences shape how agent red teaming works:

• It reads untrusted content. Emails, web pages, documents, and tool results all reach the model, so an attacker can plant instructions without talking to the agent. This is indirect prompt injection, named and demonstrated in a 2023 paper by Greshake and colleagues.
• It acts on goals. An attacker who redirects the agent’s objective turns every capability it has against its owner. The OWASP Top 10 for Agentic Applications, published in December 2025, ranks this first as agent goal hijack.
• It calls tools. A successful attack can now move money, change records, or send data out. The OWASP list calls this tool misuse and exploitation.
• It remembers. Memory and retrieved context carry over between sessions, so a poisoned entry keeps shaping decisions long after the attack. OWASP lists memory and context poisoning as its own risk.
• It works with other agents. Through the Model Context Protocol (MCP), an open standard for connecting AI applications to tools and data, and the A2A protocol for agent-to-agent messages, a failure in one agent can reach the next.

So the unit under test is the agent: its system prompt, tools, memory, permissions, and the policies it’s meant to follow. In A2A setups, the agent card, a JSON document that lists an agent’s skills and capabilities, is a useful map of what to attack.

Why multi-turn attacks matter

Most automated tests send one prompt and score one reply. Real attackers work in stages. A multi-turn attack breaks its objective into a sequence of turns, across a conversation or a chain of agent actions, with each turn building on the last so no single step looks harmful enough to trip a defense. Gradual escalation, the approach behind Crescendo, is one version of this; others split a harmful task into pieces that each look harmless on their own, or change tactics based on how the system responds.

Microsoft researchers showed how well this works with Crescendo, a multi-turn jailbreak that opens with a general question and builds on the model’s replies. They report high attack success rates, meaning the share of attempts that reach the attacker’s goal, across every model they tested.

For agents, the problem compounds. A test that stops after one turn misses failures that only appear on turn ten, after the agent has called two tools and stored something in memory.

AI red teaming vs penetration testing

The question of AI red teaming vs penetration testing comes up in most programs, because both involve attacking a system you own. Penetration testing, in NIST’s definition, is security testing that copies real-world attacks to find ways around a system’s security features. A pen test has a defined scope, a time window, and a report of exploitable weaknesses in networks, applications, and infrastructure.

The short answer to red teaming vs pentesting is the question each one asks. A pen test asks whether known weaknesses can be exploited. A red team asks what a motivated adversary could achieve.

For AI, the gap is wider. A pen test checks the infrastructure around a model, such as its APIs and cloud configuration. An AI red team tests how the model and agent behave, and the failure is often the system doing exactly what it was told by the wrong person.

Most organizations deploying agents need both, so the red team vs penetration testing question is mostly about which to run first.

Vulnerability scanning and red teaming

Scanning sits between penetration testing and red teaming. An LLM vulnerability scanner runs a library of known attacks against a model and reports which ones land. garak, NVIDIA’s open-source scanner, probes for prompt injection, jailbreaks, data leakage, hallucination, and toxic output, and Vijil researchers co-authored the paper that describes it.

Scanning is the fastest form of LLM security testing, and it belongs in every pipeline. Its limit is the library. A scan finds attacks someone has already written down, and garak’s own maintainers say it isn’t a benchmark, because its probes keep changing as new failures are found.

An AI vulnerability assessment built on scans tells you about known weaknesses. An agentic AI vulnerability assessment needs the adaptive, multi-turn search a red team brings, because an agent’s failures depend on its tools and context.

AI security testing needs all three: penetration tests for the infrastructure, scans for known attacks, and red teaming for everything an adversary could still try. For more on what scanners still catch, read the post on LLM security scanning with garak.

How an AI red teaming exercise works

A red teaming methodology for AI follows the same arc as a security engagement: plan, attack, judge, fix, repeat. The OWASP GenAI Red Teaming Guide, published in January 2025, spreads the work across model evaluation, implementation testing, infrastructure assessment, and runtime behavior analysis.

Teams coming from security will recognize this as a red team assessment methodology, adapted to systems that don’t behave the same way twice. For agents, the red team process usually runs in five steps.

1. Define the agent and its rules

Start with what the agent is for, who uses it, and what it must never do. Its system prompt, tool list, data access, and written policies become the rules an attack tries to break.

A finance-approval agent and a customer-support chatbot need different tests. A generic suite misses what matters most for each.

2. Threat model the agent

Threat modeling, a structured way to list what could go wrong and where an attacker could get in, turns those rules into attack goals. Pick a risk taxonomy so coverage can be measured.

The OWASP Top 10 for Agentic Applications is the common default risk taxonomy for constructing tests for agents. MITRE ATLAS, a public knowledge base of attacker tactics and techniques against AI systems, adds techniques drawn from real attacks.

3. Run the attacks

Mix manual and automated attacks across direct inputs and indirect channels: documents the agent retrieves, tool outputs, and messages from other agents. Run conversations rather than single prompts.

Save every transcript. The finding is the sequence of turns that led to the failure.

4. Judge the results against the agent’s policies

A transcript becomes a finding when a person, or a judge model, decides it broke a rule. Score against the agent’s own policies, and track the attack success rate per risk category so you can see what has been tested and what hasn’t.

Agents are non-deterministic, so run the important attacks more than once. One clean run says little about the next.

5. Fix, then test again

Findings feed two kinds of fix. Some change the agent itself, like a tighter system prompt or narrower tool permissions. Others become runtime controls that block the behavior in production.

Then retest. The OWASP GenAI Red Teaming Guide treats red teaming as continual oversight, and no agent stays tested once it changes.

Manual vs automated red teaming

Manual testing finds the creative attacks. Microsoft’s AI Red Team, which has red teamed more than 100 generative AI products, lists the human element among the eight lessons it drew from that work. People notice when a reply is subtly wrong or a domain rule has been bent.

Manual testing doesn’t scale. An enterprise running dozens of agents can’t hand-test each one after every model update or prompt change, so coverage gets traded against budget.

Automated red teaming closes that gap, and Microsoft’s AI Red Team lists automation as another lesson: it helps cover more of the risk space. AI red teaming tools range from scanners to attack frameworks. PyRIT, Microsoft’s open-source Python Risk Identification Tool for generative AI, automates routine red teaming tasks so testers can focus on harder ones. garak automates probe libraries.

The newer approach is autonomous AI red teaming, where attacker models plan multi-turn attacks, read the target’s replies, and adapt between attempts. It brings the persistence of a human red teamer to every release. The research note on how Diamond’s attacker agents plan multi-turn attacks shows one design in detail.

The workable answer is both. Use automation for breadth and for every release, and people for depth and judgment.

AI red teaming frameworks and standards

No single AI red teaming framework covers agents end to end. A handful of sources do most of the work.

OWASP AI red teaming guidance comes in two parts. The OWASP GenAI Red Teaming Guide gives the method, and the OWASP Top 10 for Agentic Applications gives the risks, from agent goal hijack to rogue agents.

The Cloud Security Alliance’s Agentic AI Red Teaming Guide, released in May 2025, goes further into agents. It sets out test requirements, steps, and example prompts for red teaming AI agents. MITRE ATLAS maps attacker techniques, modeled on the MITRE ATT&CK framework many security teams already use.

NIST’s Generative AI Profile (NIST AI 600-1), a July 2024 companion to the NIST AI Risk Management Framework, lists AI red-teaming among its suggested actions. One action covers testing resilience against attacks such as prompt injection and data poisoning.

Regulation is starting to name the practice too. The EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. If you deploy agents rather than build models, that duty doesn’t apply to you directly, but the question behind it will come up in audits and customer security reviews.

For AI red teaming compliance, the record matters as much as the test. Map each finding to a framework category, keep the transcripts, and note what was fixed and when. The post on operationalizing agentic AI covers how that evidence fits an ISO 42001 AI management system.

AI red teaming best practices

Most red team best practices carry over from security. The ones below are specific to agents, and they work as a red teaming checklist before your next agent ships:

• Test the agent as deployed, with its real tools, permissions, and data connections. A stripped-down copy hides the paths an attacker would use.
• Write down the agent’s policies first, so every finding points to a rule it broke.
• Cover the indirect channels: retrieved documents, tool outputs, memory, and messages from other agents.
• Run multi-turn attacks, and repeat the important ones, because a single pass of a non-deterministic system proves little.
• Track coverage against a named taxonomy, so you know what hasn’t been tested yet.
• Retest after every model update, prompt change, or new tool.
• Keep transcripts and findings inside your own environment, because they describe how to break your agent.
• Hand findings to the people who can fix them, in the tools they already use.

A transcript that breaks your agent is a working exploit, and the traces behind it often contain sensitive data. That’s one reason regulated teams run red teaming inside their own network, as the post on why the on-premise option matters explains.

Where red teaming falls short

Red teaming finds failures. It can’t prove there are none left.

A clean report means the attacks you tried didn’t work. Microsoft’s AI Red Team says as much: one of its lessons is that red teaming is different from safety benchmarking, and another is that AI safety and security will never be fully solved.

The term itself is loose. A 2024 study by Feffer and colleagues found little agreement on what AI red teaming covers or what should happen with the results. The authors warn that treating it as a catch-all answer to safety concerns verges on security theater.

Results also expire. The agent you tested isn’t the agent running after the next model update, prompt edit, or new tool.

And a red team run off to the side, by a separate team on its own schedule, delivers findings after the agent has shipped. For AI red teaming in practice, the fix is to make it part of agent evaluation. It becomes one more test in the pipeline that decides whether an agent is ready, run as often as the agent changes.

How we approach AI red teaming at Vijil

We built Vijil Diamond for AI agent verification. It compiles each agent’s purpose, personas, and policies into an agent harness, which asks the same questions on every run and scores the agent on reliability, security, and safety.

Adaptive Red Teaming, launched as part of Diamond in October 2026, is our approach to agentic red teaming. It sends an adaptive red team of attacker agents against your live agent. The attackers hold full conversations in successive waves, each starting from what the last one learned and grounded in the agent’s own tools, policies, and personas. Each finding arrives with the conversation behind it, mapped to the OWASP Top 10 for Agentic Applications or your own catalog.

It runs in your VPC or fully on premises, so transcripts stay inside your network, and it fits into CI/CD. Developers can also start it from Claude Code through a plugin.

Findings carry forward to Vijil Dome, which provides AI runtime security for agents in production. What the red team finds shapes what the agent is allowed to do.

If you’d rather have adversarial AI testing services than a tool, we can run the first one with you. Our guided pilot takes one agent through agent lifecycle management with your team, then hands over the controls.

Final thought

AI red teaming earns its place when it changes what an agent is allowed to do. A finding that stays in a report changes nothing.

For agents, that means testing the whole system, across turns, tools, and memory, every time it changes, and turning each finding into a control.

Get in touch to see how Adaptive Red Teaming works against your own agents, inside your own environment.