Research map / Large language models / Agents, code and math
Combining Induction and Transduction for Abstract Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Llama3.1-8B separately for inductive (Python program generation) and transductive (direct output prediction) ARC solving on synthetic ARC-style problems, finding them complementary and ensembling to 56.75% on ARC validation.
Problem
Few-shot abstract reasoning on ARC: whether to infer an explicit latent function (induction) or directly predict test outputs (transduction) from few input-output examples.
Approach
Meta-train the same LLM architecture and the same synthetic problem set for both induction (generating Python code checked against training examples) and transduction (direct output grid prediction), show they solve largely different problems, and ensemble them by trying induction first with transduction as fallback.
Key results
- Ensemble of induction and transduction (TTT + reranking) reaches 56.75% on ARC validation, close to the average human 60.2%.
- Induction and transduction solve largely different problems despite identical training data and architecture; solved sets are stable across random seeds.
- Smaller Kaggle-scale version scores 19% on private test and 36.5% on validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine induction with transduction as a fallback when no sampled program fits the examples | BARC on ARC-Potpourri · ARC validation | accuracy (2 tries): 43.00% -> 56.75% | Helped |
| Add test-time training to transduction (ablation) | BARC on ARC-Heavy · ARC validation | accuracy (2 tries): 19.25% -> 29.75% | Helped |
| Add test-time reranking to transduction without TTT (ablation) | BARC on ARC-Potpourri · ARC validation | accuracy (2 tries): 29.125% -> 35.25% | Helped |
| Add test-time training without reranking (ablation) | BARC transduction on ARC-Potpourri · ARC validation | accuracy (2 tries): 29.125% -> 39.25% | Helped |
| Add reranking to transduction with TTT (ablation) | BARC transduction on ARC-Potpourri · ARC validation | accuracy (2 tries): 39.25% -> 43.00% | Helped |
| Use GPT-4 rather than GPT-4o-mini to generate synthetic problem descriptions (ablation) | BARC induction · ARC validation | accuracy (2 tries): 11.07% -> 18.78% | Helped |
| Use GPT-4 rather than GPT-4o-mini to generate synthetic problem descriptions (ablation) | BARC transduction · ARC validation | accuracy (2 tries): 13.50% -> 15.25% | Helped |
| Use GPT-4 rather than GPT-4o-mini to generate synthetic data (ablation) | BARC ensemble · ARC validation | accuracy (2 tries): 19.50% -> 26.50% | Helped |
| Increase the induction sampling budget from 10k to 20k programs and use majority vote (ablation) | BARC induction · ARC validation | accuracy (2 tries): 30.50% -> 38.00% | Helped |
| Scale the system down to 336 programs, beam size 3, and no TTT (ablation) | BARC ensemble · ARC validation | accuracy (2 tries): 56.75% -> 36.5% | Hurt |
| Scale the system down to 336 programs, beam size 3, and no TTT (ablation) | BARC ensemble · ARC private test | accuracy (2 tries): 56.75% -> 19% | Hurt |
| Train induction and transduction models on the same problems using the same architecture | Induction and transduction models · ARC validation | task overlap | Mixed |
Surprising findings
- Induction and transduction are strongly complementary even when controlling for data, priors and architecture, contradicting RobustFill's finding that induction is superior.
- Models surpass human accuracy on the hardest ARC problems but underperform on the easiest ones.
Limitations
- Evaluated only on ARC; generality to other benchmarks is untested.
- Bootstraps from manually written seed programs rather than learning from solving new problems.
- Flagship model is too expensive for Kaggle private test; scaled-down version scores much lower.
Experimental setup
Datasets: ARC training split ReARC ARC validation split ConceptARC
Benchmarks: ARC validation (400 tasks) ARC private test (Kaggle) ConceptARC
Metrics: solve rate in 2 tries (pass@2) pass@3 on ConceptARC
Baselines: CodeIt Claude-3.5 ARChitects ReARC RobustFill
Builds on: ReARC CodeIt test-time training Llama 3.1
Closest papers by meaning
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.86)
- Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? (ICLR 2025 · similarity 0.85)
- PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks (NeurIPS 2026 · similarity 0.85)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.85)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.85)
- Program Synthesis via Test-Time Transduction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.