Research map / Large language models / Agents, code and math
IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IRIS, a neuro-symbolic system that uses LLMs to infer taint specifications and filter false positives on top of CodeQL for whole-repository Java vulnerability detection, evaluated on a new 120-vulnerability benchmark CWE-Bench-Java.
Problem
Static taint analysis for security vulnerabilities depends on manually curated third-party API specifications and produces many false positives, while LLMs alone cannot reason over whole repositories.
Approach
Combine LLM-inferred CWE-specific source/sink specifications (fed into CodeQL as custom queries) with LLM-based contextual classification of each detected dataflow path to remove false positives, enabling whole-project vulnerability detection.
Key results
- IRIS with GPT-4 detects 55 of 120 vulnerabilities versus 27 for CodeQL (+28), with average FDR 84.82% vs 90.03% for CodeQL.
- DeepSeekCoder 7B with IRIS detects 52 vulnerabilities, showing smaller open models are competitive.
- On latest versions of 30 Java projects, IRIS with GPT-4 found 4 previously unknown vulnerabilities not detected by CodeQL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine CodeQL static taint analysis with LLM-inferred specifications and GPT-4 contextual analysis | CodeQL · CWE-Bench-Java | #Detected: 27 -> 55 | Helped |
| Combine CodeQL static taint analysis with LLM-inferred specifications and GPT-4 contextual analysis | CodeQL · CWE-Bench-Java | Avg FDR (%): 90.03 -> 84.82 | Helped |
| Combine CodeQL static taint analysis with LLM-inferred specifications and GPT-4 contextual analysis | CodeQL · CWE-Bench-Java | Avg F1 Score: 0.076 -> 0.177 | Helped |
| Use IRIS with GPT-3.5 instead of CodeQL alone | CodeQL · CWE-Bench-Java | Avg FDR (%): 90.03 -> 90.42 | Hurt |
| Use IRIS with Llama 3 8B instead of CodeQL alone | CodeQL · CWE-Bench-Java | Avg F1 Score: 0.076 -> 0.058 | Hurt |
| Use IRIS with Llama 3 70B instead of CodeQL alone | CodeQL · CWE-Bench-Java | #Detected: 27 -> 54 | Helped |
| Use IRIS with Qwen-2.5-Coder 32B instead of CodeQL alone | CodeQL · CWE-Bench-Java | #Detected: 27 -> 47 | Helped |
| Use IRIS with Gemma-2 27B instead of CodeQL alone | CodeQL · CWE-Bench-Java | #Detected: 27 -> 45 | Helped |
| Use IRIS with DeepSeekCoder 7B instead of CodeQL alone | CodeQL · CWE-Bench-Java | #Detected: 27 -> 52 | Helped |
| Add LLM-inferred source specifications (ablation) | IRIS with GPT-4 · CWE-Bench-Java | #Detected | Helped |
| Add LLM-inferred sink specifications (ablation) | IRIS with GPT-4 · CWE-Bench-Java | #Detected | Helped |
| Add contextual analysis to filter false-positive paths (ablation) | IRIS with GPT-4, GPT-3.5, or Llama 3 70B · CWE-Bench-Java | Avg F1 Score | Helped |
Surprising findings
- Contextual LLM filtering reduced false positives only for stronger models (GPT-4, GPT-3.5, Llama-3 70B); smaller models increased false positives after filtering.
- CWE-78 (OS command injection) remains hard for all LLMs, with at most 3 of 13 detected.
Limitations
- Many vulnerabilities (e.g., side-effect-driven or non-taint-modelable ones) remain undetected.
- Multiple LLM calls for specification inference and filtering increase cost.
- Evaluated only on Java; generalization to other languages is unknown.
Experimental setup
Datasets: GitHub Advisory database CWE-Bench-Java
Benchmarks: CWE-Bench-Java
Metrics: number of vulnerabilities detected average false discovery rate average F1 detection rate
Baselines: CodeQL Facebook Infer SpotBugs Snyk
Builds on: CodeQL MERLIN Seldon
Closest papers by meaning
- SecCodePRM: A Process Reward Model for Code Security (ICML 2026 · similarity 0.87)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.86)
- ProSec: Fortifying Code LLMs with Proactive Security Alignment (ICML 2025 · similarity 0.86)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.85)
- MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents (NeurIPS 2026 · similarity 0.85)
- Security Attacks on LLM-based Code Completion Tools (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.