Research map / Large language models / Agents, code and math
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Kazuki Fujii, Yukito Tajima, Sakae Mizuki, Masaki Kawamura, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Masanari Oi, Taishi Nakamura, Takumi Okamoto, Shigeki Ishida, Kakeru Hattori and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LLM style-guided rewriting (SGCR) after syntax and lint filtering (ablation) | SwallowCode pipeline · HumanEval/HumanEval+ | pass@1: over 7-9 points | Improved |
| add self-contained optimization rewriting (SCOR) after SGCR (ablation) | SwallowCode pipeline · HumanEval/HumanEval+ | pass@1: over 5-6 points | Improved |
| add syntax error filtering (ablation) | The-Stack-v2 Python · HumanEval/HumanEval+ | pass@1 | Improved |
| add pylint filtering (threshold 7.0) (ablation) | The-Stack-v2 Python · HumanEval/HumanEval+ | pass@1: more than 1 point | Improved |
| add LLM-based score filtering over linter filtering (ablation) | The-Stack-v2 Python · HumanEval/HumanEval+ | pass@1: less than 1 point | No clear effect |
| apply SGCR after filtering instead of on raw data (ablation) | SGCR · HumanEval/HumanEval+ | pass@1: +0.4-2.1 points | Improved |
| apply SGCR (renaming functions) (ablation) | SwallowCode · MBPP | accuracy: approximately 10-point decrease | Hurt |
| replace Stack-Edu with SwallowCode | Llama-3.1-8B · HumanEval | pass@1: +17.0 | Improved |
| replace Stack-Edu with SwallowCode | Qwen2-7B · HumanEval | pass@1: 39.3 -> 49.6 | Improved |
| replace Finemath-4+ with SwallowMath | Llama-3.1-8B · GSM8K | accuracy: +12.4 | Improved |
| replace Finemath-4+ with SwallowMath | Llama-3.1-8B · MATH | accuracy: +7.6 | Improved |
| switch from Japanese-heavy to English-heavy mixture | Llama-3.1-8B · HumanEval | accuracy: 0.583 (English-heavy mixture) | Improved |
Closest papers by meaning
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.90)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks (ICML 2024 · similarity 0.89)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
