Research map / Large language models / Agents, code and math
Automatically Testing Functional Properties of Code Translation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the Nomos hyperproperty testing framework to test functional (syntactic and semantic) k-safety properties of code translation models and proposes a property-guided search over model temperature to obtain better translations.
Problem
Code translation models (LLM-based transpilers) may produce incorrect code, and existing evaluation relies on small, manually or automatically generated test suites over a small corpus, leaving generalization unclear; users also want a correct translation for a given program without retraining.
Approach
Express syntactic and semantic 1- and 2-safety (hyperproperty) specifications for transpilers in an extended Nomos language, test them via metamorphic test harnesses that generate program variants, and use a lexicographic hill-climbing search over model parameters (temperature) that minimizes violated properties for a given input program.
Key results
- Testing with beam size 1 finds 27 to 37 violated properties out of 38 across the four models, with 5564 to 11213 total violations.
- The property-guided search raises passing programs from 66 to 84 for TransCoder and from 71 to 83 for DOBF out of 100 programs.
- Violated properties for the final TransCoder translation drop from 78 to 35 (DOBF: 100 to 63) after search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search over temperature for up to 20 iterations | TransCoder · Java-to-Python, 100 programs | passing programs: 66 to 84 | Helped |
| Search over temperature for up to 20 iterations | TransCoder · Java-to-Python, 100 programs | violated properties: 78 to 35 | Helped |
| Search over temperature for up to 20 iterations | DOBF · Java-to-Python, 100 programs | passing programs: 71 to 83 | Helped |
| Search over temperature for up to 20 iterations | DOBF · Java-to-Python, 100 programs | violated properties: 100 to 63 | Helped |
| Increase beam size from 1 to 3 | code translation models · Java-to-C++ and Java-to-Python | violated properties: decreases by between 3 and 7 | Helped |
| Use StarCoder instead of specialized translation models | Java-to-Python translation · Java-to-Python | total property violations: 11213 vs 5777; 11213 vs 5564 | Hurt |
Surprising findings
- General-purpose StarCoder violates more properties (37 of 38) than the specialized translation models (30-31).
- Increasing beam size from 1 to 3 reduces violations but the number of violated other (compiles/retValues) properties decreases only marginally.
Limitations
- Evaluation limited to Java-to-C++ and Java-to-Python translation on a single 545-program LeetCode benchmark.
- Search uses only temperature as the mutated parameter and simple stochastic hill climbing on a small set of 100 programs.
Experimental setup
Datasets: TransCoder benchmark of 545 LeetCode solutions in Java, C++, and Python
Benchmarks: TransCoder 545-program LeetCode benchmark
Metrics: percentage of unique property violations number of violated properties total violations number of passing programs
Baselines: TransCoder DOBF TransCoder-IR StarCoder
Builds on: Nomos metamorphic testing TransCoder DOBF TransCoder-IR
Closest papers by meaning
- Function-to-Style Guidance of LLMs for Code Translation (ICML 2025 · similarity 0.89)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.87)
- Guess & Sketch: Language Model Guided Transpilation (ICLR 2024 · similarity 0.86)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.86)
- Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.86)
- CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.