Building a Practical Evaluation for AI Agents in Cybersecurity

Contributors: Sabarish Radhakrishnan, Andrew Hu, Kushal Borra, Sidhanth Poduri, Viraj Ganguli, and Cedric Anderson

Introduction

Artificial intelligence has moved beyond generating text. Modern agents can read files, use command-line tools, interact with applications, analyze traffic, and make changes to code. That makes evaluation harder than sending the same prompt to several models and comparing their final answers.

Over the past several months, our team has been building a project to evaluate how AI agents perform cybersecurity tasks. It started with a simple question: if we give several agents the same security problem, which one performs the best?

As we got further into the work, that question became less useful. An agent is not just a model. It also includes the harness running it, the instructions it receives, the tools and permissions it has, the provider route, the runtime environment, and the grader interpreting the result. If a run fails, the model may be responsible, but the provider may also have returned an incomplete response, the harness may have stopped early, the tool may not have reached the target, or the fixture may not have started correctly. We needed to be able to tell the difference.

The AI Agent Evaluation Project grew out of that problem. The goal was not to build the largest benchmark we could or produce a universal model ranking. We wanted a smaller cybersecurity evaluation where the cases are understood, execution is controlled, and the evidence is detailed enough to explain what happened.

That makes the project useful beyond a one-time comparison. It gives Obscurity Labs a way to decide where an agent deserves further investment, where an existing security tool remains the better choice, and where the two may work well together. It also gives us a repeatable way to test new configurations without relying on a demonstration or a single successful run.

The methodology draws on principles from the NIST AI Risk Management Framework, ISO/IEC 42001, and the OWASP AI Testing Guide. Those sources helped shape how we approached system boundaries, risk, validation, and evidence. The harder part was turning those ideas into an evaluation we could actually operate.

The Decision Comes Before The Benchmark

It is tempting to begin an evaluation by choosing models and collecting prompts. That gets a project moving quickly, but it can leave the most important question unanswered: what decision will the result support?

We ran into this early with the phrase “cybersecurity performance.” It sounds specific until you try to measure it. Reviewing source code is different from testing a running application. Both are different from analyzing a packet capture, reading a structured network log, or changing code without breaking expected behavior. A system can be useful in one setting and unreliable in another, so a broad score would tell us less than it appeared to.

We narrowed the project to a focused security problem that could be represented through several kinds of work. SQL injection is one example within that scope because the same underlying issue can appear in source code, application behavior, packet captures, and Zeek logs. The point was not to treat one vulnerability as a stand-in for cybersecurity. It gave us a stable thread while we changed the evidence, tools, and operating conditions around it.

That choice also made the business purpose clearer. We wanted to know whether a particular agent configuration could perform useful security work under defined conditions, how it compared with established tools, and what review would still be needed before using it in a real workflow. Once the decision is clear, the team can ask whether the cases resemble the work and whether the measurements would change what we do next.

Treat the Agent as a Working System

At the beginning of the project, we thought mostly in terms of comparing models. The runs quickly showed why that framing was incomplete. Harnesses decide how instructions are presented, which tools are available, how context is managed, when a run stops, and what happens when a tool fails. Providers add another layer through routing, capacity, and response behavior. The runtime determines which files, services, and network destinations the agent can reach.

We therefore define and version the complete configuration being evaluated, including:

• The provider route, harness, and instructions
• Available tools and permissions
• Context settings, retry behavior, and time limits
• Runtime dependencies and the environment around the target

The project uses Promptfoo to organize shared evaluation definitions and connect cases, prompts, schemas, and provider routes. That gives us a common way to run different configurations while retaining the identity of the system that produced each result. It does not erase differences between harnesses, which is important because those differences are part of what we are evaluating.

One of those harnesses is Colossus, a custom agent harness included in the project. Colossus is pinned to a specific release and connected through a repository-owned provider integration. It gives us another controlled execution path while preserving run identifiers, tool activity, usage, timing, and other metadata needed to reconstruct an attempt. We record the harness and version with the result rather than treating it as invisible plumbing.

The models being tested are intentionally not identified in this post. The useful public point is that a result belongs to a complete, versioned configuration. Two runs using the same model can still differ because their harnesses, tools, instructions, or limits differ. Calling both outcomes “model performance” would hide the parts of the system a company can actually change.

Build Cases Around the Work and Its Failure Modes

After defining the system, we needed cases that reflected more than straightforward success. Security work includes benign findings, incomplete evidence, unavailable tools, permission boundaries, and situations where the correct response is to stop. If every case contains a vulnerability, an agent can appear effective by reporting a problem every time. Safe controls show whether it can recognize when no finding is warranted.

The case material includes application source, running fixtures, packet captures, and Zeek logs. Several application fixtures have vulnerable and safe counterparts so the evaluation measures both detection and restraint. Some network cases are run with approved parsing tools and others without them. That distinction helps us see what the agent contributes and what depends on the surrounding toolset.

Each case has a stable identity, fixture metadata, expected behavior, and a defined output schema. A versioned manifest links the selected cases, prompt variants, fixtures, and expected assignments for an approved run. This matters because repositories change. Without a manifest, adding a prompt or fixture to a directory could quietly change the next run while leaving the report looking comparable to the last one.

Ground truth stays outside the environment available to the agent. Depending on the case, a grader may check the verdict, source location, request parameter, packet location, structured output, or required tool use. We also include smaller checks for instruction following and basic tool behavior because they help explain more complicated failures. Identifying the right issue but returning an unusable result is different from missing it entirely.

The project does not rely only on agent comparisons. Conventional security tools run against the same fixtures where the comparison is meaningful. Static-analysis baselines include language-appropriate tools, while dynamic and network baselines use established scanners and traffic-analysis tools. In practice, that includes technologies such as Semgrep, GoSec, Ruff, ESLint, SQLMap, ZAP, Suricata, Snort, Zeek, and TShark across the relevant portions of the suite.

Those baselines are not included to set up a replacement contest. They give us a reference point and help test the evaluation itself. If a scanner cannot start, a ruleset cannot load, or a target cannot be reached, we investigate the fixture and runner before making a claim about an agent.

Test the Evaluation Before Trusting the Results

One of the less obvious lessons from this work is that the evaluation itself needs validation. A broken fixture, an unreachable service, or a grader that rewards the wrong behavior can produce a clean-looking result. Running more models does not correct that problem. It only repeats it at greater cost.

Before a controlled run begins, preflight checks that:

• The selected manifest, cases, and fixture paths agree
• Target applications can start and pass their health checks
• Prompts, schemas, provider routes, and harness versions match the run plan
• Required tools, runtimes, and artifact locations are available
• Execution limits, repository state, and spending controls are in place

Some of these checks are intentionally offline. Others require a live provider route or a running fixture. The report keeps those states separate so a successful configuration check is not presented as proof that a live evaluation completed. Paid execution is also tied to the approved configuration. Material changes to the cases, routes, limits, or harnesses require another review.

The graders are tested with correct, incorrect, partial, malformed, and adversarial output. We prefer deterministic checks when the task allows them, including schema validation, comparison with ground truth, and inspection of resulting artifacts. When success depends on what happened in the environment, the grader checks that outcome rather than trusting the agent’s explanation. Claiming a target was tested is not evidence that the tool reached it.

We start with repository validation and bounded end-to-end runs before scaling. This has caught errors in fixture setup, output handling, tool availability, and grading before they consumed a larger run. The procedure is not especially glamorous, but it has been one of the most useful parts of the project.

Report What Happened, Not Just Who Scored Highest

A single score is attractive because it makes comparison easy. It is also easy to misuse. During an agent run, a provider can be unavailable, a dependency can be missing, a fixture can fail its health check, a policy can deny an action, or the agent can finish without a valid final response. Those outcomes have different owners and different remedies.

We separate failures according to where they occurred:

• Model quality, such as a wrong or malformed answer
• Provider availability, such as a capacity or routing error
• Evaluation-platform execution, such as a fixture or scanner failing to start
• Policy and containment, such as an action being intentionally denied

A scanner that never started is an operational failure and does not become evidence that the target is safe. Invalid or ambiguous cases are held for review rather than silently changed after the outcome is known.

The report retains more than the final answer. Depending on the harness and task, the evidence can include tool calls, commands, file changes, intermediate events, timing, usage, and errors. Promptfoo provides the evaluation structure, Colossus and the other harness integrations supply their execution records, and repository-owned tooling normalizes the information needed for grading and reporting.

This evidence lets a reviewer see whether the agent reached the right target, used the expected method, and arrived at a conclusion the artifacts support. It also makes conventional metrics more meaningful. True positives, false positives, true negatives, and false negatives are useful only when the underlying cases ran correctly and the denominator is honest.

For Obscurity Labs, the result can inform research priorities, tool selection, regression testing, and the controls needed before higher-consequence use. It may show that an agent is promising but operationally unreliable, or that a traditional tool has stronger detection while the agent provides more useful context. The purpose is not to force a winner between them.

Because the cases, manifests, harnesses, and results are versioned, the evaluation can be rerun as the technology changes. Validated incidents and reviewer corrections can become regression cases. That gives us a way to track improvement on work we care about instead of producing a report that ages with the first set of systems it tested.

The Team Behind the Project

The AI Agent Evaluation Project was developed as a collaborative Obscurity Labs research and engineering effort. The work was supported by company leadership, including Alexander Rymdeko-Harvey, Co-Founder and Chief Executive Officer; Keelyn Roberts, Co-Founder and Chief Operating Officer; and Tory Clasen, Chief Technology Officer.

Viraj Ganguli and Cedric Anderson provided the primary direction and technical management for the internship work. They helped connect individual implementation tasks to the larger evaluation, reviewed technical decisions, and kept the focus on reproducibility, execution controls, and evidence that could be independently reviewed.

Through Obscurity Labs' participation in the Commonwealth Cyber Initiative internship program, Sabarish Radhakrishnan, Andrew Hu, Kushal Borra, and Sidhanth Poduri contributed to the implementation, testing, documentation, fixture setup, security-tool baselines, containment, evaluation workflows, validation, and research planning behind the project.

The work took place through the same shared repository, review, and validation process used across the project. The intent was not to produce several disconnected demonstrations, but to leave behind a system that Obscurity Labs can continue to use and extend.

Whitepaper

Standard Operating Procedure: AI Agent Evaluation

https://github.com/obscuritylabs/ai_eval_sop

Conclusion

There are several directions this work could take, including additional vulnerability classes, new security workflows, repeated trials, stronger cross-platform containment, and broader remediation coverage. We do not plan to add those simply to make the benchmark larger. Each addition needs ground truth we can defend, an environment we can control, and a clear reason for including it.

The lesson from this project is that useful agent evaluation depends on understanding the path from the task to the evidence. If we cannot explain what the agent saw, what it could reach, what actually ran, and why the grader reached its conclusion, then the score is not ready to support much of a decision.

Our objective is straightforward: build an evaluation that is specific enough to mean something and honest enough to remain useful when the result is not what we expected.