Research map / Adversarial robustness and security / Certified and empirical robustness

RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

NeurIPS 2026 · Adversarial robustness and security › Certified and empirical robustness · approach: Provable defenses and their limits · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RustMizan, a compilable, multi-level Rust CVE benchmark with four vulnerability-analysis tasks and a semantics-preserving mutation framework for contamination testing and adversarial robustness, evaluated on four frontier LLM agents.

Problem

Existing vulnerability benchmarks use small non-compilable snippets, focus on binary detection, and risk training-data contamination; Rust is underrepresented.

Approach

A Rust vulnerability benchmark where each CVE is packaged as compilable crate, file, and function-level variants with four task annotations, paired with a compiler-tool-based mutation framework (rustfmt, syn/quote, rust-analyzer) that preserves ground truth via marker, content-based, and rename-validation tracking.

Key results

• Four frontier models reach 56-65% CVC detection accuracy but only 17-23% line-localization F1.
• Function-level context raises line F1 relative to crate level for all four models (e.g., Claude 17.9 to 39.4).
• Malignant (adversarial comment/rename) mutations drop mean line F1 by 5.6 points (about 27% relative), while benign and Rust-specific mutations change it by under 1 point.

Surprising findings

• Benign and Rust-specific mutations leave line F1 essentially unchanged, so no aggregate contamination signal is found for vanilla models.
• Gemini 3.1 Pro was observed explicitly recalling a CVE identifier and line numbers from memory in one trajectory.

Limitations

• Small dataset of 42 CVEs and 173 variants.
• Pre-patch code treated as vulnerable and post-patch as non-vulnerable, which may not hold.
• Uneven mutation coverage across variants; partial mutation in 29 of 173 variants.

Experimental setup

Datasets: RustSec Advisory Database

Benchmarks: RustMizan

Metrics: CVC Accuracy CWE F1 Function localization F1 Line localization F1 Success@1

Baselines: CVE-Bench CodeMorph DEFECTS4J-TRANS ReCode VJBench-trans

Builds on: ReAct CVE-Bench CodeMorph DEFECTS4J-TRANS ReCode

Closest papers by meaning

• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.87)
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.86)
• CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.85)
• SecCodePRM: A Process Reward Model for Code Security (ICML 2026 · similarity 0.85)
• CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.84)
• Automated Proof Generation for Rust Code via Self-Evolution (ICLR 2025 · similarity 0.84)