Research map / Large language models / Agents, code and math
CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CircuitSeer, which selects SFT data by scoring each problem with the attention variance of a small set of identified reasoning heads, then soft-samples 10% of OpenR1-Math for fine-tuning.
Problem
Large reasoning SFT datasets are expensive to train on, and existing data selection methods rely on costly external models or heuristic quality/difficulty criteria.
Approach
Identify reasoning-critical attention heads via ablation (loss increase on a probe set) in a reference model, then score each training problem by the variance of the mean attention over those heads on the problem input alone, and soft-sample proportional to that score.
Key results
- On Qwen2.5-Math-7B-Base, fine-tuning on 10% CircuitSeer-selected data gives average Pass@1 64.5 vs 61.2 for random and 63.1 for full dataset (+1.4 over full).
- On Qwen2.5-Math-7B-Base, CircuitSeer improves Majority@16 on AIME24 by 10.0 points over random (63.3 vs 53.3).
- Reasoning-head scores beat all-heads scores by 3.3 average P@1 points on Qwen2.5-Math-7B (64.5 vs 60.0 for All Heads and 61.2 for Random in the ablation table).
Surprising findings
- Input-only scoring (problem text, no generated solution) outperforms output-based scoring (64.5 vs 62.8 average P@1).
- Top-k hard selection underperforms random on three of nine benchmarks, while soft sampling over the same scores does not.
Limitations
- Evaluated mainly on math reasoning with a single data source (OpenR1-Math), so generality to other domains is untested.
- Relies on a reference model and a probe set of 300 high-loss samples; the head-selection procedure's sensitivity is not fully characterized.
- Reported gains on many benchmarks are small (often a few points) and based on single runs for most settings.
Experimental setup
Datasets: OpenR1-Math-220K
Benchmarks: AIME24 AIME25 AMC23 MATH500 OlympiadBench Kaoyan GAOKAO 2023 GAOKAO-Math GAOKAO 2024
Metrics: Pass@1 Majority@16
Baselines: Random Diversity Quality Max Loss IFD
Builds on: Transformer circuits undifferentiated attention ablation IFD
Closest papers by meaning
- A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning (NeurIPS 2025 · similarity 0.88)
- Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data (AAAI 2024 · similarity 0.88)
- Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties (NeurIPS 2025 · similarity 0.88)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.87)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.87)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.