Research map / Large language models / Language, safety and interpretability
Towards Practical Defect-Focused Automated Code Review
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a repository-aware, merge-request-level automated code review pipeline (code slicing, multi-role LLM reviewers with filtering, inline line localization) evaluated on historical C++ fault-inducing merge requests at an industrial recommendation service.
Problem
Automated code review is usually framed as snippet-level code-to-text generation scored by BLEU-like metrics, which ignores repository context, merge-request-level evaluation, and defect detection.
Approach
Combines AST-based code slicing for repository context, a multi-role LLM system (Reviewer, Meta-Reviewer, Validator, Translator) with CoT prompts, a Q1-Q3 scoring comment filter against nitpicks/hallucinations, and inline line-number prompt formatting, evaluated with defect-oriented MR-level metrics (KBI, FAR, CPI, LSR).
Key results
- The framework reaches up to about 10x higher KBI/CPI than prior baselines (CodeReviewer, CCT5, LLaMA-Reviewer, DISCOREV), which score near 0-2% KBI.
- LLaMA3.1-405B (quantized) with Left Flow slicing, three reviewers, meta-reviewer and validator reaches CPI1 of 22.07 with KBI 20.00 and FAR1 75.37.
- Inline line-number formatting raises KBI from 30.37 to 37.04 in the All setting and LSR to about 91%; Left Flow outperforms Full Flow in the single-reviewer setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Original Diff slicing with Left Flow slicing (ablation) | LLaMA3.1-405B-AWQ-Int4 single reviewer, all comments · Fault merge requests | KBI: 23.70 -> 37.04 | Helped |
| use Full Flow instead of Left Flow slicing (ablation) | LLaMA3.1-405B-AWQ-Int4 single reviewer, all comments · Fault merge requests | KBI: 37.04 -> 39.26 | No clear effect |
| add a Validator (ablation) | Left Flow, three-reviewer workflow · Fault merge requests | CPI_1: 17.51 -> 22.07 | Helped |
| add a Validator (ablation) | Left Flow, three-reviewer workflow · Fault merge requests | KBI: 31.11 -> 20.00 | Hurt |
| increase reviewer count from one to three (ablation) | Left Flow, +Meta Reviewer · Fault merge requests | KBI: 26.67 -> 31.11 | Helped |
| increase reviewer count from one to three (ablation) | Left Flow, +Meta Reviewer · Fault merge requests | CPI_1: 20.57 -> 17.51 | Hurt |
| add chain-of-thought prompting (ablation) | Left Flow, single reviewer, Top-5 · Fault merge requests | CPI_1: 11.10 -> 14.26 | Helped |
| add chain-of-thought prompting (ablation) | Original Diff, single reviewer, Top-5 · Fault merge requests | CPI_1: 13.05 -> 12.48 | Hurt |
| apply coarse filtering (ablation) | Left Flow, single reviewer · Fault merge requests | CPI_1: 9.77 -> 12.80 | Helped |
| apply Top-5 truncation instead of keeping all comments (ablation) | Left Flow, single reviewer · Fault merge requests | CPI_1: 9.77 -> 14.26 | Helped |
| provide inline line numbers instead of no line-position information (ablation) | three-reviewer workflow with Meta Reviewer · Fault merge requests | CPI_1: 12.21 -> 17.51 | Helped |
| provide relative line positions instead of no line-position information (ablation) | all-comments setting · Fault merge requests | KBI: 30.37 -> 42.96 | Helped |
Surprising findings
- Left Flow slicing outperforms the more detailed Full Flow, which the authors attribute to distraction from longer context.
- Validator reduces false alarms but also reduces key bug recall, and smaller models sometimes match larger ones within the framework.
Limitations
- Evaluation is on a proprietary C++ dataset of historical fault MRs; generality to other languages is not demonstrated.
- Filter thresholds are heuristic and no formal user study of perceived review quality was conducted.
Experimental setup
Datasets: historical fault reports and fix merge requests from company (C++ short-video recommendation codebases)
Benchmarks: internal fault merge request dataset
Metrics: key bug inclusion (KBI) false alarm rate (FAR) comprehensive performance index (CPI) line localization success rate (LSR)
Baselines: CodeReviewer CCT5 LLaMA-Reviewer DISCOREV
Builds on: Aider CodeReviewer
Closest papers by meaning
- RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing (ICML 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.84)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.83)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.83)
- CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (ICLR 2025 · similarity 0.83)
- ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract) (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.