Research map / Large language models / Agents, code and math
Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes NeMo-Minitron-8B (and Llama-3.2-3B) on RE-ARC with augmentations, generates candidates with threshold-based DFS sampling, and re-ranks them with a product-of-experts score over 16 augmentations, reaching 71.6% on the ARC-AGI public eval set.
Problem
Abstract reasoning on ARC-AGI remains very hard for LLMs; the goal is to get high accuracy with open, reproducible, low-cost inference.
Approach
Use one fine-tuned LLM both as a generator and a scorer: threshold-based DFS produces high-probability candidates under augmented views of each task, and candidates are re-scored by the product of their likelihoods across all augmentations (a geometric-mean product of experts), with a theorem bounding its KL divergence.
Key results
- Achieves 71.6% (286.5/400) 2-guess accuracy on the ARC-AGI public evaluation set with NeMo-Minitron-8B, the top open-source result reported.
- DFS with T=9% reaches 76.0% candidate coverage and PoE selection raises score from 66.6% (sum of probabilities) to 71.6%.
- Llama-3.2-3B reaches 61.4% with the full pipeline versus 14.9% baseline; Sudoku reaches 53% top-1 accuracy on 1000 puzzles.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add test-time training (ablation) | ARC method with NeMo-Minitron-8B · ARC-AGI public evaluation | 2-guess accuracy: 18.3\% -> 44.5\% | Helped |
| Add 16 augmented candidate samples (ablation) | ARC method with NeMo-Minitron-8B after TTT · ARC-AGI public evaluation | 2-guess accuracy: 44.5\% -> 62.5\% | Helped |
| Add product-of-experts candidate selection (ablation) | ARC method with NeMo-Minitron-8B after 16xAug · ARC-AGI public evaluation | 2-guess accuracy: 62.5\% -> 67.6\% | Helped |
| Add DFS candidate generation (ablation) | ARC method with NeMo-Minitron-8B after PoE · ARC-AGI public evaluation | 2-guess accuracy: 67.6\% -> 71.6\% | Helped |
| Add product-of-experts selection instead of selecting by summed probabilities (ablation) | NeMo-Minitron-8B with DFS T=9% · ARC-AGI public evaluation | 2-guess accuracy: 66.6\% -> 71.6\% | Helped |
| Replace PoE selection with maximum augmentation probability (ablation) | NeMo-Minitron-8B with DFS T=9% · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 63.5\% | Hurt |
| Replace PoE selection with minimum augmentation probability (ablation) | NeMo-Minitron-8B with DFS T=9% · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 68.8\% | Hurt |
| Increase DFS threshold from T=9% to T=0.5% (ablation) | NeMo-Minitron-8B · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 71.8\% | No clear effect |
| Increase DFS threshold from T=9% to T=20% (ablation) | NeMo-Minitron-8B · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 70.0\% | Hurt |
| Replace DFS with greedy candidate generation (ablation) | NeMo-Minitron-8B and the same 16-augmentation scoring · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 67.6\% | Hurt |
| Replace DFS with stochastic sampling (4x) (ablation) | NeMo-Minitron-8B and the same 16-augmentation scoring · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 70.8\% | Hurt |
| Replace DFS with beam search (4x) (ablation) | NeMo-Minitron-8B and the same 16-augmentation scoring · ARC-AGI public evaluation | 2-guess accuracy: 71.6\% -> 71.6\% | No clear effect |
Surprising findings
- Adding a few extra tokens at the start of a task slightly improved performance, which the authors attribute to a computational buffer.
- Product-of-probabilities selection outperformed sum, min and max aggregation across all sampling methods.
Limitations
- Relies on ARC-specific augmentations (D8 symmetries, color permutations, example reordering), which do not transfer directly to other domains.
- Threshold T is a hyperparameter that depends on model and training, and DFS speed depends on model certainty.
Experimental setup
Datasets: RE-ARC ARC-AGI public evaluation set ConceptARC Sudoku 3M
Benchmarks: ARC-AGI public evaluation set ConceptARC Sudoku 3M
Metrics: 2-guess accuracy top-1 accuracy runtime VRAM
Baselines: TTT BARC TTT+BARC o3 o1-preview Beam search Stochastic sampling Greedy sampling
Builds on: LoRA RE-ARC BARC test-time training
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.88)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.88)
- ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (Student Abstract) (AAAI 2025 · similarity 0.88)
- ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (AAAI 2025 · similarity 0.88)
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.