Research map / Large language models / Reasoning and chain of thought

CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CodeChemist, a training-free test-time scaling framework that transfers functional knowledge from Python to low-resource or complex-syntax languages via synthesized test oracles, switching between in-language voting and cross-lingual pass-rate selection based on an uncertainty estimate.

Problem

CodeLLMs perform much worse on low-resource programming languages (e.g., Lua) and complex-syntax languages (C++, Java) than on Python, and existing test-time scaling methods offer limited gains there due to data scarcity.

Approach

Use multi-temperature hedged sampling for target-language candidates, cluster them by execution and AST similarity to estimate uncertainty, and when uncertain build I/O oracles by running high-resource (Python) reference programs on synthesized test inputs, then select the candidate with the highest pass rate.

Key results

• On MultiPL-HumanEval Lua with Qwen2.5-Coder-1.5B, CodeChemist reaches 57.8 Pass@1 vs 34.1 vanilla and 50.9 for S*.
• On Ag-LiveCodeBench-X Lua with Qwen-32B, CodeChemist achieves 33.1 vs 21.5 vanilla.
• Compared with S*, CodeChemist uses roughly 2x fewer tokens and is about 4x faster on Qwen-32B (per the paper's efficiency analysis).

Surprising findings

• Using a weaker Qwen-3B model to generate high-resource reference code for Qwen-32B still improved Lua performance from 79.5 to 81.4.
• Most incorrect transferred test cases stem from incorrect Python reference programs, which target languages also tend to fail on, so the method is bounded by base model problem-solving ability.

Limitations

• Bounded by the base model's problem-solving ability; if the high-resource reference is wrong, transfer fails.
• Input-output behavior may not capture non-functional properties such as performance or language-specific semantics.
• Evaluation focuses on function-level generation; repository-level results are only a single case study.

Experimental setup

Datasets: MultiPL-HumanEval MultiPL-MBPP Ag-LiveCodeBench-X

Benchmarks: MultiPL-HumanEval MultiPL-MBPP Ag-LiveCodeBench-X

Metrics: Pass@1

Baselines: Majority Voting LLM Judge Self-Debugging S* CodeT MBR-Exec

Builds on: S* CodeT MultiPL-E Self-Debugging

Closest papers by meaning

• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
• Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck (NeurIPS 2026 · similarity 0.89)
• Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.89)
• T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Models (ICLR 2026 · similarity 0.88)
• EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.88)
• AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.88)