Research map / Large language models / Agents, code and math
DRPBench: Evaluating LLMs in Concurrent Code Comprehension via Fine-Grained Data Race Prediction
Yuqi Guo, Siwei Wei, Yan Cai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable test-time reasoning (thinking mode) | Gemini 3 · DRPBench-Lite | F1: ↑17.70% | Improved |
| enable test-time reasoning (thinking mode) | DeepSeek-V3.2 · DRPBench-Lite | F1: ↑23.80% | Improved |
| enable test-time reasoning (thinking mode) | Claude Sonnet 4.5 · DRPBench-Lite | F1: ↓11.73% | Hurt |
| enable test-time reasoning (thinking mode) | Qwen3-32B · DRPBench-Lite | F1: ↑31.85% | Improved |
| enable test-time reasoning (thinking mode) | Llama3-70B · DRPBench-Lite | F1: ↑27.45% | Improved |
| enable test-time reasoning (thinking mode) | Qwen2.5-32B · DRPBench-Lite | F1: ↑8.39% | Improved |
| run inside OpenHands agentic scaffold instead of standard prompting | Gemini 3 · DRPBench-Lite | F1: ↑4.15% | Improved |
| run inside OpenHands agentic scaffold instead of standard prompting | DeepSeek-V3.2 · DRPBench-Lite | F1: ↓4.44% | Hurt |
| run inside OpenHands agentic scaffold instead of standard prompting | Claude Sonnet 4.5 · DRPBench-Lite | F1: ↓19.12% | Hurt |
| use agentic scaffold instead of reasoning mode | Gemini 3 · DRPBench-Lite | F1: ↓13.56% | Hurt |
| use agentic scaffold instead of reasoning mode | DeepSeek-V3.2 · DRPBench-Lite | F1: ↓28.24% | Hurt |
| use agentic scaffold instead of reasoning mode | Claude Sonnet 4.5 · DRPBench-Lite | F1: ↓7.39% | Hurt |
Closest papers by meaning
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.88)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
