Research map / Large language models / Agents, code and math
UniCode: Augmenting Evaluation for Code Reasoning
Xinyue Zheng, Haowei Lin, Shaofei Cai, Yaodong Yang, Zilong Zheng, Yitao Liang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from seed problems to augmented variants | state-of-the-art LLMs · UniCode augmented problems | Pass@1: 31.2% performance collapse | Hurt |
| apply narrative perturbation | LLMs · UniCode narrative axis | Pass@1: o4-mini 92.5%; deepseek-r1 94.0% | No clear effect |
| apply efficiency scaling (larger input size) | o4-mini · UniCode efficiency axis | Pass@1: 37% drop | Hurt |
| apply sequential composition | gemma-3-27b-it · UniCode sequential axis | Pass@1: 1.3% | Hurt |
| alter rules, composition or concept (reasoning graph topology) | all models · UniCode | Pass@1 | Hurt |
| add brute-force filtering, majority vote and LLM adjudication to test generation | UniCode test pipeline · Test-Eval | test correctness: 94.5% correctness, 86.0% coverage | Improved |
| add adversarial and corner test cases beyond random ones | evaluated LLMs · UniCode | Pass@1 drop: qwen3-235b-a22b Corn -13.3% | Hurt |
| use reasoning models instead of non-reasoning | LLMs · UniCode | Pass@1: 70.3% vs 49.8% best | Improved |
| increase augmentation depth | LLMs · UniCode | Pass@1 | Hurt |
| use different generator models for augmentation (ablation) | UniCode · model rankings | ranking | No clear effect |
Closest papers by meaning
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.87)
- CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance (ICML 2025 · similarity 0.87)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.87)
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
