AI Paper Atlas

Research map / Learning theory and algorithms / Combinatorial and search optimization

How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective

Xianzhen Luo, JinYang Huang, Wenzhen Zheng, Qingfu Zhu, Mingzheng Xu, Yiheng Xu, YuanTao Fan, Wanxiang Che

ICLR 2026 · Learning theory and algorithms › Combinatorial and search optimization · approach: Predict-then-optimize learning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a method with correct-code outputs (LCB) instead of direct generation (CRUX) (ablation)Qwen2.5-Coder-32B · TC-BenchHackRate: 18.73 -> 58.10Improved
switch base model from Qwen2.5-Coder-32B to Deepseek-V3 with LCB (ablation)LCB · TC-BenchHackRate: 58.10 -> 58.83No clear effect
generate many easy test cases (higher PassRate) with CRUX versus ALGO (ablation)Qwen2.5-32B · TC-BenchHackRate: PassRate 26.71 vs 20.48; HackRate 15.43 vs 21.96Hurt
use TC-Bench basis instead of all wrong codes (ablation)LCB on Claude-4-Thinking · 100-problem subsetHackRate: near 100% -> just over 50%Improved
random sampling 20 wrong codes (TestCase-Eval) instead of the diverse basis (ablation)evaluation set · 100-problem subsetscore inflationNo clear effect
limit to 5 wrong codes (TCG) instead of rank-sized basis (ablation)evaluation set · 100-problem subsetHackRateHurt
increase number of test cases from 1x to 2x problem rank (ablation)test case generation · TC-BenchHackRateImproved
increase test cases from 3x to 5x rank (ablation)test case generation · TC-BenchHackRateNo clear effect
add more correct codes for validating generated tests (ablation)test validity filtering · TC-BenchHackRateMixed
add input validator (HT) over LCB (ablation)LCB · TC-BenchHackRate: Claude4 62.08 -> 62.96; Qwen2.5-Coder-32B 58.10 -> 43.76Mixed
use PSEUDO (majority vote of 10 solutions) instead of ALGO (one brute force) (ablation)test case generation · TC-BenchHackRate: Claude4-Thinking 35.46 -> 14.02Mixed
filter wrong codes with failure rate above 80% and problems with all-ones columns (ablation)WrongSelect pre-filtering · raw submissionsinflation resistance: removes 13% of raw WCs, about 5% of problemsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy