How to find agent AI risks with black-box red teaming

Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery is a study released as arXiv 2609.09647, and the original is linked under the title. It proposes one connected framework that finds multi-step behavioral risks of agentic AI from a black-box position, because it combines a seven-domain taxonomy with an automatic scenario generator called SAGE-RT and with judge model evaluation checked against human judgment.

What interests me more than the headline numbers is the shift in perspective behind them, since most safety evals still stare at a single exchange inside a chat window. Real agents, however, live outside that window, so in this post I will walk through why that gap exists and how this research tries to close it.

Hello, I am Patrick, and let us start outside the chat window

Hello, I am Patrick.

Imagine a new colleague joins your team, yet this colleague does not only talk, because this person also reads email, runs payments, and sometimes deletes files. So would you evaluate this colleague on eloquence alone?

I found myself asking the next question at once, since modern agents work much like that colleague, as they read untrusted inputs, call tools with real permissions, and choose their own next actions. That combination widens the security surface far beyond a chat only model, so the paper takes this working picture as its starting point and treats autonomous tool use as normal rather than exceptional.

Why a single question and answer cannot show an agent

Readers who trust chat benchmarks may push back here, because they may ask why a one-turn question is not enough and why we must watch several steps in a row. The answer is that agent failures rarely explode inside one sentence, since they look polite in step one and only slip near the third or fourth tool call.

Take an expense settlement agent on a Monday morning, for it first reads email, then opens a receipt file, and then calls a payment tool. When a strange email in the middle gets interpreted as an instruction, the final payment step can then produce a completely different result. A single-turn test sheet never captures that chain, because the chain only exists across steps, so when you run the whole trajectory instead, you finally see where the judgment slid. I nodded along at this point, because the paper moves the unit of evaluation from the sentence to the full trajectory.

What black box means in this context

So what does black box mean here? The phrase appears often in security writing, yet it points at something slightly different each time. That is why the definition in this study needs to be stated plainly first, and the definition turns out to be refreshingly modest.

Black box here means testing only from the outside without going inside, since the tester does not open weights, does not read the original system prompt, and does not ask for privileged logs. What the tester holds instead is only a basic description of the system, so a run might start from one or two lines such as this agent handles customer requests and uses a refund tool. Of course an insider could learn more by looking inside, but for an outside auditor or an adopting company the blind position is the realistic one. I liked this setup, because it turns a field constraint into the test condition instead of assuming laboratory privileges.

What changes when risks are split into seven domains

An interesting question follows, namely why split risk into seven at all, and what changes just because we draw boundaries. The answer sits inside the phrase observable behavior, because the taxonomy connects what can be seen from outside to a risk category without guessing at inner intent.

If that sounds abstract, think of a medical checkup, since a fever alone says little about where the illness sits, while temperature plus blood pressure plus blood work makes a response much clearer. This taxonomy does the same work, for in the governance domain behavior such as skipping an approval step or leaving no trace stands out. In the privacy domain, the visible pattern is then information that should have stayed hidden flowing toward another agent, while in the agent behavior domain a different cluster appears around tool misuse or self-willed goal changes. Once the risks are separated this way, the result is no longer a vague score but a diagnosis that points at where to fix. The number seven may sound large, yet the measured failures break differently by domain, so merging them would only blur the picture.

How 120 scenarios per domain ever get printed

The problem comes next, since dividing risk into seven domains is a fine idea, but if people must handwrite every test for each domain, the work never ends. That is why the study brings out an automatic generator called SAGE-RT, and this generator prints 120 adversarial scenarios for every domain.

A printing press is a useful metaphor here, because each domain definition in the taxonomy acts as a plate, and the generator turns that plate while twisting the situation a little at a time. In governance that variation then accumulates into cases that bend an approval flow, cases that coax the agent to leave no audit trail, and cases that blur who is responsible. With 120 items across seven domains the full exam thus reaches 840 items. That volume is punishing for handwriting, yet it is a natural batch for a machine that stamps structured variations from seeds. A fair worry is whether machine printed items collapse into near duplicates with light rewording, but the measured risk differs by domain, which eases that worry indirectly, because identical items would have produced a flat scoreboard.

If tests are automatic, who grades them

The next doubt arrives immediately, since who grades 840 test sheets when reading them one by one is far too much for people. So the study seats judge models, or LLM judges, as graders, and it then checks separately whether that grading deserves trust by comparing it with human judgment.

This can sound circular at first, because a machine writes the questions and another machine grades them, so readers may suspect mutual favoritism. The human validation step exists exactly for that reason, since the team first confirms that judge scores point in the same direction as human readings before handing over large scale grading. Learning works the same way for people, as a trainee first compares answers with a teacher, and only after the rubric matches does the trainee grade alone. That does not mean the judge is perfect, but it does mean the team avoided grading 840 sheets against a rubric that had already drifted from human judgment. I read that procedure as a minimum safety rail rather than a proof of perfection.

What the two stages of CrewAI and AutoGen showed

From a systems perspective, the next question is obvious, namely could this exam work only inside one particular structure. To answer that, the team mounted the same tests on two different agent architectures, CrewAI and AutoGen, and it also rotated across four base models while keeping the exam fixed.

That choice deserves a second look, because a diagnostic that shines on only one architecture stays a half tool in production. When the same test sheet bites on two structures with different ways of dividing roles, the sheet is then probably not overfitted to one of them. Rotating across four base models serves the same purpose, since it separates quirks of one model voice from problems that come from the agentic way of working itself. This is where I would be careful not to over-read the experiment in the optimistic direction, because if the result survives as a structural problem rather than one model fault, then swapping models alone will not fix it.

What the 56.25 percent governance number really says

Now it is time to look at the numbers, but numbers need their conditions read alongside them, since the 56.25 percent average governance risk looks at first like merely a high score, yet its meaning only sharpens when the scenes behind it are read together.

Governance risk is, in plain terms, about procedure and traces, because skipping a step that needed approval, or acting in a way that leaves later reviewers unable to tell who did what and why, counts as an organizational accident. So 56.25 percent means the agent broke such procedure in more than half of the tested scenes. Because the figure repeats across many scenarios rather than one or two slips, it thus deserves extra attention. Not every violation turns directly into a large accident, but when procedure wobbles in most scenes, an operations team cannot relax. I think the more important result is the structure behind the number, since an instruction to follow procedure was probably written somewhere in a prompt, yet it faded during multi-step execution.

Why privacy leaks more in multi-agent setups

So how should we read the 65 percent privacy risk in multi-agent configurations? Working together leaking more than working alone can feel backward at first. Once the way agents work is pictured, though, the result quickly makes sense, and this section shows the back side of collaboration.

When several people work together, one remark can spread across the whole team, and agent teams are not very different, because information read by one agent slips into the context handed to another. The moment agent A includes something from a customer record inside a summary for agent B, knowledge that B never needed then starts to spread. After two or three such handoffs, the boundary of who may know what thus turns blurry. Sharing is the job itself in multi-agent structures, so a vague order to stop sharing cannot fix it. I read that 65 percent less as a breach rate and more as the cost of collaboration, since the more work is divided, the easier it becomes to divide what should never have been shared, and that tension looks built into agent collaboration.

The weight of saying agent behavior breaks 85 percent of the time

This is the number to handle with the most care, because an 85 percent vulnerability rate in agent behavior is so large that the first suspicion is whether the exam was unfairly harsh. That suspicion is healthy, so this is where I think readers should slow down before trusting or dismissing the figure.

Agent behavior means handling tools and holding onto goals, since calling the wrong tool, or dropping the assigned task to chase a side objective, already counts as failure. The tests in this domain press exactly on that joint, as strange inputs shake tool choice and intermediate goals overwrite the original goal, and those scenes repeat with persistence. When such scenes break at rates reaching 85 percent, the reading is then that judgment tilts easily once the agent holds tools in hand. Not every break turns into real damage, but the opening an attacker can choose from is clearly wide. When my team looks at a new AI architecture we check the operating cost behind the score as well as the score, so in that spirit I would call this 85 percent closer to a warning light than a grade.

What to actually change in practice

A practical question remains, because the deployment meeting still opens on Monday morning after the paper is closed. I translated what this research leaves behind into three scenes, and those scenes feel surprisingly actionable.

First, there is now a shared language for pre-deployment checks, since with seven domains as a common form, builders and buyers can talk across the same table. They can then walk down where governance stood, where privacy leaked, and which behavior bent, and adjust access rights or contract terms. Second, outside testing becomes feasible, because runs need only a basic description rather than internal code, so auditors or customers can run the exam themselves. Third, testing can repeat, so when the model or the architecture changes, the same 840 item exam runs again and the before and after can be compared. None of this means the framework catches every risk. Still, moving work that used to be felt out by hand into something a machine runs systematically should change the morning of operations a little. What I would watch next is the human part that remains, namely deciding what must be protected and which failures can never be tolerated.