Research map / Large language models / Agents, code and math
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PotARCin, a five-dimension (definition, classification, constrained generation, editing, inversion) generator/verifier-based benchmark extending ARC-AGI-1 evaluation, plus the hand-crafted P-ARC test set, showing frontier models' full-task accuracy falls 25–52 points below output-grid accuracy.
Problem
Standard ARC evaluation only checks whether a model outputs the correct grid for a test input, which may overstate whether models have acquired the underlying transformation rule as a reusable skill.
Approach
Probe five uses of each ARC task's inferred rule (definition, classification, constrained generation, editing, inversion) with generator and verifier programs and corrupted pairs, scoring full-task accuracy across all five dimensions.
Key results
- Full-task accuracy on ARC-AGI-1 training is 21-58%, a 25-52 point drop from output-grid accuracy of 63-89%.
- On P-ARC, full-task accuracy falls to 1-8% for all five models.
- Conditioned on a correct Definition, agreement with the model's own answer is 99.1% when the answer is correct but 21.1% when incorrect (ARC-AGI-1).
Surprising findings
- Kimi K2.5 has the lowest output-grid accuracy yet nearly doubles MiniMax's full-task accuracy, reordering models ranked similarly by standard accuracy.
- Probing five dimensions once exposes a failure in 25.5% of trials versus 20.2% when a single dimension is sampled five times.
Limitations
- Generator and verifier programs may capture nearby variants rather than the intended rule; ARC tasks are underspecified so Classification labels can be contestable.
- ARC-AGI-1 training results may be inflated by training-data exposure; reasoning budgets differ across models (Claude 120K thinking budget).
Experimental setup
Datasets: ARC-AGI-1 training set ARC-GEN stable dataset RE-ARC H-ARC
Benchmarks: ARC-AGI-1 training set P-ARC
Metrics: full-task accuracy per-dimension accuracy independence baseline Wilson confidence intervals
Baselines: GPT-5.4 Gemini 3.1 Pro Claude Opus 4.6 Kimi K2.5 MiniMax M2.5
Builds on: Potemkin understanding RE-ARC ARC-GEN V-ARC
Closest papers by meaning
- Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning (NeurIPS 2026 · similarity 0.90)
- Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective (ICML 2025 · similarity 0.87)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.87)
- Towards Generative Abstract Reasoning: Completing Raven’s Progressive Matrix via Rule Abstraction and Selection (ICLR 2024 · similarity 0.86)
- From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (NeurIPS 2026 · similarity 0.86)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.