Research map / Learning theory and algorithms / Combinatorial and search optimization
How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Xianzhen Luo, JinYang Huang, Wenzhen Zheng, Qingfu Zhu, Mingzheng Xu, Yiheng Xu, YuanTao Fan, Wanxiang Che
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a method with correct-code outputs (LCB) instead of direct generation (CRUX) (ablation) | Qwen2.5-Coder-32B · TC-Bench | HackRate: 18.73 -> 58.10 | Improved |
| switch base model from Qwen2.5-Coder-32B to Deepseek-V3 with LCB (ablation) | LCB · TC-Bench | HackRate: 58.10 -> 58.83 | No clear effect |
| generate many easy test cases (higher PassRate) with CRUX versus ALGO (ablation) | Qwen2.5-32B · TC-Bench | HackRate: PassRate 26.71 vs 20.48; HackRate 15.43 vs 21.96 | Hurt |
| use TC-Bench basis instead of all wrong codes (ablation) | LCB on Claude-4-Thinking · 100-problem subset | HackRate: near 100% -> just over 50% | Improved |
| random sampling 20 wrong codes (TestCase-Eval) instead of the diverse basis (ablation) | evaluation set · 100-problem subset | score inflation | No clear effect |
| limit to 5 wrong codes (TCG) instead of rank-sized basis (ablation) | evaluation set · 100-problem subset | HackRate | Hurt |
| increase number of test cases from 1x to 2x problem rank (ablation) | test case generation · TC-Bench | HackRate | Improved |
| increase test cases from 3x to 5x rank (ablation) | test case generation · TC-Bench | HackRate | No clear effect |
| add more correct codes for validating generated tests (ablation) | test validity filtering · TC-Bench | HackRate | Mixed |
| add input validator (HT) over LCB (ablation) | LCB · TC-Bench | HackRate: Claude4 62.08 -> 62.96; Qwen2.5-Coder-32B 58.10 -> 43.76 | Mixed |
| use PSEUDO (majority vote of 10 solutions) instead of ALGO (one brute force) (ablation) | test case generation · TC-Bench | HackRate: Claude4-Thinking 35.46 -> 14.02 | Mixed |
| filter wrong codes with failure rate above 80% and problems with all-ones columns (ablation) | WrongSelect pre-filtering · raw submissions | inflation resistance: removes 13% of raw WCs, about 5% of problems | Improved |
Closest papers by meaning
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.87)
- TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling (NeurIPS 2026 · similarity 0.85)
- rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.85)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
