AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Themis: Automated Constraint-Aware Test Synthesis Framework for Code Reinforcement Learning

Shengyu Ye, Qi Liu, Hao Jiang, Zheng Zhang, Heng Yu, Zhenya Huang

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Themis, a constraint-aware automated test synthesis framework that generates higher-quality test suites to provide more reliable reward signals for RL-based code generation training of LLMs.

Problem

RL for code generation in LLMs depends on high-quality test suites for reward signals, but current test suites have insufficient quantity and quality, causing false positives (incorrect solutions passing) and slow positives (valid but suboptimal solutions passing slowly).

Approach

Constraint-aware automated synthesis of test cases for code RL, aimed at reducing both false positives and slow positives in verification rewards. Details beyond the abstract are not available.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
RL with Themis-augmented tests instead of original TACO testsQwen2.5-Coder-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 57.9 -> 59.4Improved
RL with Themis tests instead of no RLQwen2.5-7B · Average of LCB, HumanEval, MBPPPass@1 (%): 41.1 -> 56.7Improved
RL with Themis tests vs original TACO testsQwen2.5-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 55.3 -> 57.1Improved
RL with original TACO tests on already strong instruct modelQwen2.5-Coder-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 57.2 -> 57.9No clear effect
training on random KodCode 10k (false-positive-prone tests) (ablation)Qwen2.5-Coder-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 57.2 -> 56.7Hurt
filter KodCode samples with few tests or low difficulty (ablation)Qwen2.5-Coder-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 56.7 -> 57.7Improved
augment filtered KodCode with Themis tests (ablation)Qwen2.5-Coder-7B-Instruct · Average of LCB, HumanEval, MBPPPass@1 (%): 57.7 -> 58.8Improved
Themis test reduction (about 5x compression) (ablation)Themis tests · TACO-verified error detection (DeepSeek-R1 generator)error detection rate (%): 95.6 -> 94.7No clear effect
Themis tests vs EvalPlus-style testsTest generation · TACO-verified error detection (DeepSeek-R1 generator)error detection rate (%): 66.2 -> 95.6Improved
Themis tests vs original TACO testsTest generation · TACO-verified error detection (DeepSeek-R1 generator)error detection rate (%): 89.3 -> 95.6Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy