Research map / Large language models / Agents, code and math
CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CS-Bench, a quadrilingual (EN/ZH/FR/DE) benchmark of about 10K samples across 26 computer science subfields and 4 tasks, evaluates over 30 LLMs, and studies links between CS, math, and coding ability.
Problem
Existing LLM benchmarks either cover CS only as a small slice of broad multi-discipline exams or focus on specific skills such as math and coding, leaving no comprehensive evaluation of computer science knowledge and reasoning.
Approach
A multilingual CS benchmark with four task formats (multiple-choice, assertion, fill-in-the-blank, open-ended) and knowledge vs. reasoning labels, paired with analyses of scale-score fitting, error types, and correlations between CS and math/code performance.
Key results
- Overall zero-shot scores range from 39.86% to 72.29%, with GPT-4o highest at 72.29% and Claude-3.5-Sonnet at 71.07%.
- Score trends across CS-Bench correlate with Math (GSM8K, MATH) and Code (HumanEval, MBPP) benchmarks with Pearson coefficients above 0.9 (up to 0.96 on MBPP).
- Scale-score fitting from 7B-70/72B models predicted Qwen1.5-110B at 67.83% versus actual 67.95%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 5-shot answer-only prompting instead of 0-shot | Models evaluated on CS-Bench · CS-Bench (EN) | average score: +1.47% | Helped |
| Use 5-shot COT prompting instead of 0-shot | Models evaluated on CS-Bench · CS-Bench (EN) | average score: +2.00% | Helped |
| Specialize DeepseekLLM-7B for mathematics | DeepseekLLM-7B · CS-Bench (EN) | overall Avg score (%): 43.67 -> 50.49 | Helped |
| Specialize Llama2-7B for code with CodeLlama | Llama2-7B · CS-Bench (EN), Data Structure and Algorithm | knowledge score (%): 51.51 -> 58.90 | Helped |
| Specialize Llama2-7B for code with CodeLlama | Llama2-7B · CS-Bench (EN), Computer Organization | overall score (%) | Hurt |
| Increase model scale within the same model family | LLMs evaluated on CS-Bench · CS-Bench | overall score | Helped |
| Increase Qwen1.5 scale from 72B to 110B | Qwen1.5 · CS-Bench (EN) | overall Avg score (%): 2.66% | Helped |
| Use OpenAI-o1-mini reasoning | OpenAI-o1-mini · CS-Bench reasoning problems | score: +11.97 points | Helped |
| Use OpenAI-o1-preview reasoning | OpenAI-o1-preview · CS-Bench knowledge and reasoning questions | score | Helped |
| Use OpenAI-o1-preview reasoning instead of GPT-4o | CS-Bench reasoning questions · CS-Bench reasoning questions | token consumption: 21.24 times | Hurt |
| Specialize InternLm2-7B for mathematics with InternLm-Math-7B | InternLm2-7B · CS-Bench (EN), Computer Network | reasoning score (%): 40.60 -> 44.47 | Helped |
| Specialize InternLm2-7B for mathematics with InternLm-Math-7B | InternLm2-7B · CS-Bench (EN) | overall Avg score (%): 51.56 -> 48.45 | Hurt |
Surprising findings
- Qwen1.5-7B outperforms Llama2-70B on GSM8K and MATH, yet Llama2-70B beats Qwen1.5-7B on CS-Bench, showing math benchmark rank does not transfer to CS.
- OpenAI-o1-preview consumes about 21 times the tokens of GPT-4o on reasoning questions, while only modestly improving scores.
Limitations
- Covers only university-level CS fundamentals across four domains; does not cover research-level CS or production scenarios.
- Generative tasks (about 20% of samples) are scored by GPT-4, which may introduce evaluation bias and cost.
- Multilingual coverage limited to English, Chinese, French, and German.
Experimental setup
Datasets: GSM8K MATH HumanEval MBPP GPQA C-Eval MMLU
Benchmarks: CS-Bench
Metrics: accuracy GPT-4-scored score (0-1 mapped) Pearson correlation
Baselines: GPT-4o GPT-4 Claude-3.5-Sonnet Claude-3-Opus Llama3.1-405B Mistral-Large-123B Qwen1.5-110B Llama3-70B
Builds on: MMLU C-Eval DeepSeekMath CodeLlama
Closest papers by meaning
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.90)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
- PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts (NeurIPS 2025 · similarity 0.90)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.90)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.