Research map / Large language models / Agents, code and math

LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

NeurIPS 2026 · Large language models › Agents, code and math · approach: Code generation benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LibEvoBench, a version-indexed benchmark over PyTorch, NumPy, and SciPy API evolution plus the SEUS metric, finding frontier LLMs are largely version-oblivious on evolving APIs.

Problem

LLMs must generate code that uses the correct API version of a library, but their training on temporally mixed corpora causes anachronistic errors where APIs from a different library version are called.

Approach

A deterministic, version-indexed API compatibility matrix (built from Sphinx inventories and runtime introspection) enables a three-task benchmark and a hallucination taxonomy that separates anachronistic predictions from fabrication, combined into a stability- and anachronism-penalized score (SEUS).

Key results

• GPT-5.4 achieves the top SEUS of 86.0 (stable 92.0, evolving 86.4), with GPT-5.5 close behind at 85.1.
• Providing relevant redacted documentation yields a consistent 10-20 point gain in API-calling accuracy, while adding an explicit version constraint yields no measurable gain.
• Even the strongest models lose 5-10 points on evolving APIs relative to stable APIs, and Qwen3.5 scale (35B to 397B) shifts quality up without changing the degradation pattern.

Surprising findings

• Explicitly specifying the target library version provides no measurable improvement across any model family.
• Scaling Qwen3.5 from 35B to 397B does not reduce the evolving-API degradation curve, suggesting the gap is structural rather than capacity-limited.

Limitations

• Possible contamination of API-C samples mined from public repositories that may appear in training corpora; direct contamination probes infeasible for closed models.
• Evaluation uses zero thinking budget, so results reflect parametric knowledge without extended reasoning.
• Covers only three Python libraries (PyTorch, NumPy, SciPy) and 9-10 versions each.

Experimental setup

Datasets: GitHub public repositories with requirements.txt pins Sphinx objects.inv documentation inventories runtime package introspection of PyTorch, NumPy, SciPy versions

Benchmarks: LibEvoBench

Metrics: exact match parameter recall parameter F1 SEUS retention anachronism rate

Baselines: GPT-4.1 GPT-5 GPT-5.1 GPT-5.4 GPT-5.5 Gemini 2.0 Flash Gemini 2.5 Flash Gemini 3 Flash Claude Sonnet 4 Claude Sonnet 4.6 Qwen3.5

Builds on: GitChameleon CodeUpdateArena VersiCode LibEvolutionEval

Closest papers by meaning

• CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.91)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.87)
• Toward Interactive Understanding of Code APIs (NeurIPS 2026 · similarity 0.87)
• Simple Baselines are Competitive with Code Evolution (NeurIPS 2026 · similarity 0.86)
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.85)
• Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.85)