Research map / Multi-agent systems and games / LLM agents
QiMeng-LibBench: Benchmarking LLM Agents for Library-Scale Cross-Architecture Migration
Weijia Li, KE GAO, Jiajie Li, Han Sun, Yuhe Ding, Yongdong Mai, Yiran Le, Yongjie Qian, Zhibin Zhang, Xinyu Wang, Limin Cheng, Shouxu Kuang and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QiMeng-LibBench, a benchmark for evaluating LLM agents on library-scale cross-architecture migration of high-performance code with ISA-specific SIMD intrinsics.
Problem
Cross-architecture migration of high-performance libraries requires handling library-scale dependencies and performance-critical SIMD kernels, but existing LLM agent evaluations are limited to function-level tasks or scalar code.
Approach
A benchmark targeting library-scale migration including ISA-specific SIMD intrinsics, assessing agents on both migration correctness/speed and peak performance rather than function-level scalar code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPT-5 as the underlying LLM instead of the second-best model | agents · QM-LibBench | SR@1: average +10.39 SR@1 | Improved |
| use open-weights Qwen3-coder-480b instead of proprietary models | OpenHands · QM-LibBench | SR@1: 17.65 overall SR@1 | Hurt |
| pair Gemini-3-pro-preview with the weaker SWE-Agent instead of OpenHands | Gemini-3-pro-preview · QM-LibBench | overall SR@1: 36.47 -> 3.53 | Hurt |
| migrate to emerging ISAs (RISC-V, LoongArch) instead of x86->ARM | LLM agents · QM-LibBench | average accuracy: 31.37% -> 18.14% (RISC-V), 20.59% (LoongArch) | Hurt |
| migrate OpenCV instead of ncnn | LLM agents · QM-LibBench | average SR: 23.37% (ncnn) vs 14.68% (OpenCV) | Hurt |
| raise speedup threshold p from 0 to 1.0 | all agent-LLM combinations · QM-LibBench | fastp | Hurt |
| use OpenHands instead of JoyCode with GPT-5 | OpenHands · QM-LibBench | cost on successful migrations: up to 3x longer, 4x more API calls | Hurt |
| use OpenCode instead of other agents with GPT-5 | OpenCode · QM-LibBench OpenCV | SR@1 and cost: 33.33 SR@1, time 429, money 1.75 | Improved |
| failed trajectories vs successful ones | SWE-Agent, JoyCode, OpenHands · QM-LibBench | money cost (SR@1): OpenHands 2.04 -> 2.19 | Hurt |
Experimental setup
Benchmarks: QiMeng-LibBench
Metrics: migration correctness migration speed peak performance
Closest papers by meaning
- Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench (NeurIPS 2026 · similarity 0.85)
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.84)
- ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management (ICML 2026 · similarity 0.83)
- Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces (ICML 2025 · similarity 0.83)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.83)
- Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
