mneme — Multi-Agent Vulnerability Research System with Structured Causal Memory

mneme solved 1,372 of 1,507 CyberGym Level 1 tasks in a single trial, achieving 91.0% pass@1. The CyberGym baseline using the same underlying model (Claude Opus 4.6) is 66.6%. The 24.4 percentage-point gap suggests that multi-agent orchestration, domain-specific analysis tools, and structured memory can add substantial capability beyond the foundation model alone.

CyberGym is a benchmark of 1,507 real-world vulnerability reproduction tasks derived from OSS-Fuzz across 188 open-source software projects. At Level 1, the agent receives a vulnerability description and unpatched source code, then must produce a PoC that triggers on the pre-patch build but not on the post-patch build.

System Architecture

mneme (Greek: Μνήμη, "memory") is a multi-agent security analysis system that discovers, analyzes, and reproduces software vulnerabilities, powered by a single foundation model with structured causal memory. It is built on the premise that vulnerability reproduction is a memory-retrievable task: once the causal structure of a vulnerability class is distilled into abstract policies, a new instance of the same class can be solved by retrieving the right policy and grounding it in the current codebase. Its key differentiator is verification-gated, scope-isolated causal memory.

mneme is a general-purpose vulnerability research agent, not one built specifically for this benchmark — its analysis pipeline, memory architecture, and tool skills are designed for broad applicability to real-world vulnerability discovery. All vulnerability analysis, PoC construction, and iterative refinement are performed through coordinated scout and worker agents. Every byte of every PoC is authored by the model's own reasoning.

mneme operates in two architecturally separated pipelines. The Learning Pipeline uses workers to analyze codebases, extract abstract traces, and consolidate verified knowledge into OKF memory. The Running Pipeline (used for this evaluation) measures current ability using read-only OKF — memory is consumed but never written during a trial.

How mneme Finds Vulnerabilities

mneme operates as a coordinated multi-agent system. A scout agent performs initial reconnaissance on the target — analyzing source code structure, identifying input entry points, tracing data flow to potentially vulnerable sinks, and retrieving relevant knowledge from OKF. The scout's structured findings are then distributed to worker agents, each of which independently attempts to discover, analyze, and reproduce the vulnerability.

Workers are equipped with binary analysis tool skills for deep inspection: angr for static reachability analysis, construct for declarative binary format building, gdb for dynamic debugging, hexview for structured binary inspection, and pwntools for low-level payload construction.

Analysis Pipeline

1. Target reconnaissance — the scout examines source code, identifies input contracts, locates potential vulnerability sinks, and retrieves matching OKF policies.
2. Vulnerability analysis — workers analyze the target function, input format, data flow path to the sink, and the invariant that the vulnerability violates.
3. PoC construction — workers select a construction strategy based on available context: mutating existing format samples, building inputs from scratch for complex binary formats, or using bounded fuzzing to search the input space.
4. Iterative verification — each candidate is tested against the target binary with sanitizer instrumentation. On failure, workers perform coverage-directed diagnosis and iterate with alternative hypotheses.

PoC Construction Strategies

Knowledge Base in Open Knowledge Format (OKF)

Open Knowledge Format (OKF) is a knowledge representation standard by Google Cloud — a directory of markdown files with YAML frontmatter, designed to be readable without specialized tools, parseable by agents without proprietary SDKs, diffable in version control, and portable across organizations. mneme adopts OKF as the storage format for its entire knowledge base.

The knowledge base was accumulated by analyzing C/C++ codebases across more than 100 open-source projects for vulnerabilities. It was constructed entirely before the evaluation and remained read-only throughout the trial — no entries were created or modified during solving. Each entry stores only class-level abstract knowledge: no raw PoC bytes, no exact offsets, no task IDs. An automated leak audit enforces this invariant before every commit. The knowledge base contains 3,867 entries:

An internal audit found no task-level overlap between the memory-learning materials and CyberGym; OKF coverage is maintained separately.

Experimental Setup

Results

The clean score excludes 35 tasks after conservative self-auditing: 30 where a pre-existing reproducer was found baked into the Docker image, and 5 where result metadata contained fix/diff/post-patch evidence. The raw score is retained for audit.

Failure Analysis

Of the 1,507 tasks, 1,403 were solved before clean exclusions (93.1% raw score). Among the 104 unsolved tasks, the dominant residual failure mode is no_crash — the agent produced a syntactically valid input that passed format gates but failed to violate the specific invariant at the sink within a single trial.

Clean Score Protocol

The reported score uses a reviewer-conservative exclusion policy:

• Excluded: tasks where a pre-existing reproducer was harvested from the Docker image; tasks with strict fix/diff/post-patch evidence in result metadata.
• Not excluded: tasks where the candidate PoC was written to /tmp/poc (normal harness behavior); tasks where server output recorded fix_exit or target_match (grading outputs, not solve inputs).

Compliance

• No internet-assisted solving: the model attempted WebSearch (224 calls) and WebFetch (253 calls) across 29 worker agents during the trial, but all attempts were blocked at the local server level. The server_tool_use field in all 168,411 API responses recorded web_search_requests: 0 and web_fetch_requests: 0. No external data was received by any worker.
• OKF leak audit passed: no raw PoC bytes, offsets, or checksums found in memory.
• Single-attempt discipline enforced: no post-submit repair from official feedback.

Conclusion

mneme solves 1,372 of 1,507 CyberGym Level 1 tasks in a single trial (91.0% pass@1). Starting from the Claude Opus 4.6 baseline of 66.6%, mneme achieves a 24.4 percentage-point improvement through the combination of structured OKF memory, binary analysis tooling, and a multi-agent solve pipeline. Future work will apply this system to open-source zero-day research.