Research map / Large language models / Agents, code and math
CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CodeSync, a data engine that mines real API signature updates and GitHub invocations, and CodeSyncBench, a 3,300-case benchmark plus 2,200-sample update-aware training set, showing LLMs struggle to sync with API evolution even after DPO/ORPO/SimPO/SFT-LoRA.
Problem
LLMs trained on static data generate outdated third-party Python library API invocations after APIs change, producing non-executable or suboptimal code; there is no authentic real-world benchmark for updating LLMs to API changes.
Approach
Automatically track API signature changes across real library versions, retrieve real GitHub invocations via AST-based localization, and use DeepSeek-V3 to synthesize paired legacy/updated invocations, forming a benchmark and update-aware training data for knowledge-updating methods.
Key results
- Claude-3.5-Sonnet and GPT-4o score below 20 BLEU on the CCT task, showing LLMs struggle with updated APIs.
- DPO, ORPO, and SimPO improve BLEU on CCT for Qwen2.5-Coder-7B-Instruct from 5.89 (original) to roughly 21-24, and SimPO gives the best CodeBLEU at 39.39 on CCT.
- Fine-tuned 6.7B-8B models reach BLEU of 23.86-31.59 on CCT, comparable to some proprietary models, but absolute scores remain low.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply knowledge updating methods instead of using the original model | Open-source LLMs evaluated on CodeSyncBench · CodeSyncBench CCT and ECT | BLEU | Helped |
| fine-tune DeepSeek-Coder-6.7B-Instruct | DeepSeek-Coder-6.7B-Instruct · CodeSyncBench MCQ | P@1: 31.25 -> 7.88 | Hurt |
| combine SFT with RAG | Qwen2.5-Coder-7B-Instruct · CodeSyncBench CCT | CodeBLEU: 31.17 -> 40.70 | Helped |
| combine SFT with RAG | Qwen2.5-Coder-7B-Instruct · CodeSyncBench ECT | CodeBLEU: 36.00 -> 51.35 | Helped |
| increase training examples from 5 to 10 per API | Qwen2.5-7B-Instruct · CodeSyncBench CCT | API update recall | Helped |
| increase training examples beyond 10 per API | Qwen2.5-7B-Instruct · CodeSyncBench CCT | performance | No clear effect |
| train on updated method APIs rather than function or initializer APIs | Qwen2.5-7B-Instruct using knowledge updating methods · CodeSyncBench CCT | performance | Hurt |
| apply knowledge updating methods | DeepSeek-Coder-6.7B-Instruct · HumanEval | P@1: 72.56 -> 71.34; 72.56 -> 70.12; 72.56 -> 68.29 | Mixed |
Surprising findings
- Retrieval-augmented generation alone underperforms DPO on CodeBLEU in the API update setting, due to a hit rate of only about 60%.
- DeepSeek-Coder-6.7B-Instruct's MCQ performance collapses after fine-tuning, indicating degraded instruction following.
Limitations
- Method APIs are harder to update, and the pipeline only captures statically inferable variable types.
- Absolute scores remain low, so existing knowledge-updating methods are insufficient.
- Invocation collection is slow and limited to APIs with at least 15 valid invocations.
Experimental setup
Datasets: GitHub Code Search retrieved Python invocations DeepSeek-V3 synthesized legacy-updated invocation pairs HumanEval
Benchmarks: CodeSyncBench HumanEval
Metrics: BLEU ROUGE-L Relative Edit Distance CodeBLEU Pass@k
Baselines: SFT-LoRA DPO ORPO SimPO RAG GPT-4o Claude-3.5-Sonnet Gemini-1.5-Pro DeepSeek-V3 DeepSeek-R1
Builds on: DPO ORPO SimPO
Closest papers by meaning
- LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models (NeurIPS 2026 · similarity 0.91)
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.89)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.