Research map / Large language models / Agents, code and math
Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FG-LAME, a multi-agent, solution-graph-driven pipeline that generates textbook-grounded MCQ benchmarks (ML, corporate finance, personal finance) with competency metadata, and shows better coverage, lower ground-truth error, and lower contamination than prior benchmarks.
Problem
Existing foundation model benchmarks have sparse coverage of domain competencies, provide mainly aggregate scores without fine-grained metadata, are costly to curate, and are vulnerable to contamination.
Approach
Generate MCQs from textbook chapters with a designer agent (Gemini 3.1 Pro) and verifier agent (Claude Opus 4.6); problems are built from a solution graph before the question is written, so verification reduces to checking steps, and each item is tied to a two-level taxonomy of areas and competencies derived from textbook structure.
Key results
- Expert inspection found 3/100 incorrect ground-truth labels on a random ML sample and 0/80 and 0/22 on the Finance random samples, versus documented 5% GSM8K and 6.5% MMLU error rates.
- Generated benchmarks reach normalized competency entropy of 0.979 (ML), 0.961 (personal finance), and 0.997 (corporate finance), above prior benchmarks (0.60-0.85).
- In ML and corporate finance the generated benchmarks achieve the highest separability (0.108 and 0.091), and in personal finance the highest difficulty (0.501).
Surprising findings
- Under TS-Guessing, the generated benchmarks show substantially lower guessing rates than prior benchmarks, but the authors note some residual guessing success that they attribute to patterns in the options.
Limitations
- Problems are restricted to multiple-choice format, which may not capture all reasoning forms.
- Quality is bounded by the frontier LLMs used for generation and verification, and ground-truth errors are reduced but not eliminated.
- Benchmarks inherit perspectives and biases of source textbooks.
Experimental setup
Datasets: Foundations of Machine Learning Probabilistic Machine Learning: An Introduction Understanding Deep Learning Corporate Finance (Welch) Strategic Financial Planning over the Lifecycle Humanity's Last Exam (AI/CS exemplars)
Benchmarks: FG-LAME ML FG-LAME Corporate Finance FG-LAME Personal Finance MMLU-Pro ML subset Humanity's Last Exam AI subset xFinBench FinanceMath BizBench
Metrics: accuracy normalized entropy difficulty (error rate of best model) separability (mean absolute deviation) Spearman rank correlation TS-Guessing success rate ground-truth error rate
Baselines: MMLU-Pro Humanity's Last Exam xFinBench FinanceMath BizBench
Builds on: Bloom's taxonomy Humanity's Last Exam DyVal TS-Guessing AutoGen
Closest papers by meaning
- Unearthing Skill-level Insights for Understanding Trade-offs of Foundation Models (ICLR 2025 · similarity 0.87)
- Risk Aware Benchmarking of Large Language Models (ICML 2024 · similarity 0.86)
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.85)
- How Benchmark Prediction from Fewer Data Misses the Mark (NeurIPS 2025 · similarity 0.85)
- FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.