Research map / Large language models / Retrieval and knowledge
InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InfiBench, a 234-question free-form Stack Overflow QA benchmark for code LLMs scored with model-free metrics (keyword matching, blank filling, unit tests, ROUGE similarity), and evaluates over 100 code LLMs.
Problem
Existing code benchmarks focus on code generation, are saturating, share sources, and may be contaminated, leaving free-form coding question answering under-evaluated; model-based judging is costly and biased.
Approach
A benchmark of expert-curated, paraphrased Stack Overflow questions with per-question correctness criteria evaluated by four model-free metric types (keyword matching, blank filling, unit testing, dialogue similarity), validated against human judgments as an alternative to GPT-4o judging.
Key results
- GPT-4-0613 achieves 70.64% ± 0.82% on InfiBench (best@10, T=0.2), the top score among 100+ evaluated models; Codestral-22b reaches 62.98% as the best open-source model.
- Model-free InfiBench evaluation agrees with human experts on 85.1% of clear-preference pairs versus 77.8% for GPT-4o-based evaluation.
- Copying the 10 highest-voted Stack Overflow answers scores 65.18%, and fine-tuning Codellama-13b-Instruct with decontaminated Stack Overflow data raised its score from 46.37% to 60.74%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfiBench's model-free evaluation instead of GPT-4o as judge | InfiBench response-pair evaluation · InfiBench human agreement | agreement rate: 77.8% -> 85.1% | Helped |
| Use 10 highest-voted Stack Overflow answers rather than the highest-voted answer | Stack Overflow answer baselines · InfiBench | score: 56.28% -> 65.18% | Helped |
| Use the highest-voted answer rather than the officially accepted answer | Stack Overflow answer baselines · InfiBench | score: 52.90% -> 56.28% | Helped |
| Increase sampling temperature from T=0.2 to T=1.0 (ablation) | GPT-4 best@10 · InfiBench | score: 70.64% ± 0.82% -> 76.15% ± 0.21% | Helped |
| Fine-tune Code Llama 7B on Python data (ablation) | CodeLlama-7b · InfiBench | score: 37.62% ± 1.28% -> 32.89% ± 0.45% | Hurt |
| Instruction fine-tune Code Llama 7B (ablation) | CodeLlama-7b · InfiBench | score: 37.62% ± 1.28% -> 35.15% ± 1.28% | Hurt |
| Fine-tune Code Llama 13B on Python data (ablation) | CodeLlama-13b · InfiBench | score: 41.66% ± 0.84% -> 41.31% ± 0.90% | Hurt |
| Instruction fine-tune Code Llama 13B (ablation) | CodeLlama-13b · InfiBench | score: 41.66% ± 0.84% -> 46.37% ± 1.26% | Helped |
| Fine-tune Code Llama 34B on Python data (ablation) | CodeLlama-34b · InfiBench | score: 47.36% -> 43.13% | Hurt |
| Instruction fine-tune Code Llama 34B (ablation) | CodeLlama-34b · InfiBench | score: 47.36% -> 50.45% | Helped |
| Fine-tune Code Llama 70B on Python data (ablation) | CodeLlama-70b · InfiBench | score: 40.60% -> 40.29% | Hurt |
| Instruction fine-tune Code Llama 70B (ablation) | CodeLlama-70b · InfiBench | score: 40.60% -> 42.82% | Helped |
Surprising findings
- Python-specialized fine-tuning of Code Llama improves HumanEval but lowers InfiBench scores, indicating capability imbalance.
- Open-source models' scaling trend appears to flatten beyond about 40B parameters, contrary to a common linear scaling law.
Limitations
- Score is subjective since criteria are expert-annotated without a universal standard
- Benchmark is small (234 questions) due to high labelling cost; some languages have few questions
- Focuses on correctness only and ignores naturalness, safety, latency and similar aspects
Experimental setup
Datasets: Stack Overflow posts dump
Benchmarks: InfiBench HumanEval
Metrics: best@10 score keyword matching score unit test pass rate blank filling score ROUGE-based dialogue similarity agreement rate with human experts
Baselines: GPT-4-0613 GPT-4-turbo-1106 GPT-4o-2024-05-13 Claude 3 Opus GPT-3.5-turbo-0613 Codestral-22b DeepSeek Coder V2 Code Llama
Builds on: HumanEval DS-1000 LLM-Blender
Closest papers by meaning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- Code-Style In-Context Learning for Knowledge-Based Question Answering (AAAI 2024 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.88)
- Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks (ICML 2024 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.