Research map / Large language models / Agents, code and math
IndustryCode: A Benchmark for Industry Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IndustryCode, a hierarchical benchmark of 125 industrial main problems (579 sub-problems) across finance, automation, aerospace and other domains and four languages (Python, C++, MATLAB, Stata), and evaluates 20 proprietary and open LLMs, with Claude 4.5 Opus leading at 68.1% sub-problem and 42.5% main-problem accuracy.
Problem
Existing code benchmarks are confined to single domains and languages and do not test the numerical precision, multi-language breadth, and multi-step decomposition needed for real industrial code generation.
Approach
A benchmark built from industrial production code, manually reconstructed and made harder, organized as main problems decomposed into sub-problems evaluated incrementally with cumulative context, spanning multiple industrial domains and programming languages, with numerical test cases plus an LLM judge.
Key results
- Claude 4.5 Opus achieved the top overall accuracy of 68.1% on sub-problems and 42.5% on main problems.
- Qwen3-Max reached 70.4% on C++ sub-problems, above GPT-5.2 (67.9%) and Gemini-3-pro (66.0%).
- Enabling thinking mode gave average gains of about 4.70% on sub-problems and 7.65% on main problems for benefiting models, but occasionally reduced performance.
Surprising findings
- Enabling explicit thinking mode sometimes hurt performance, driven by over-engineering and context confusion.
- Open-weights Qwen3-Max beat several proprietary models on C++ sub-problems.
Limitations
- Main problems show a large drop from sub-problem accuracy, indicating weakness in multi-step orchestration.
- Stata and MATLAB results are based on very few problems, giving noisy estimates.
Experimental setup
Datasets: IndustryCode
Benchmarks: IndustryCode
Metrics: Pass@1 accuracy
Baselines: Claude 4.5 Sonnet Claude 4.5 Haiku GPT-5.2 GPT-5.1 Gemini-3-pro Gemini-2.5-pro Qwen3-Max DeepSeek-V3.2-Exp
Builds on: SciCode HumanEval SWE-bench
Closest papers by meaning
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.85)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.85)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.85)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.85)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.84)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.