Research map / Large language models / Agents, code and math
ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConceptSearch, an LLM-driven function-search (FunSearch-style) method for ARC that guides program evolution with concept-based scoring (CNN embeddings or LLM-generated natural-language hypothesis embeddings) instead of Hamming distance.
Problem
Solving ARC tasks with LLMs is hard; in program search with binary success signals, pixel-level Hamming distance gives misleading guidance, so a better scoring signal is needed to guide LLM-based program search efficiently.
Approach
Score candidate programs by comparing concept embeddings of their input-output transformations to those of the demonstration examples, using either a CNN trained on ConceptARC concept classes or an LLM-written natural-language hypothesis embedded with a fine-tuned SentenceTransformer; these scores drive parent sampling in a FunSearch-like island search over the ARC-DSL.
Key results
- LLM-based concept scoring solves 29/50 evaluation tasks versus 25/50 for Hamming distance and 13/50 for GPT-4 direct prompting.
- Concept-based scoring reduces mean iterations: LLM-based 2.05 vs Hamming 3.70 on commonly solved tasks; LLM-based is 24.7% more efficient than Hamming distance.
- CNN-based concept scoring reaches 25/50, 29.3% more efficient than Hamming distance on tasks solved by both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GPT-4 direct-grid prompting with ConceptSearch using Hamming distance | ARC task solving · ARC evaluation set (50 tasks) | accuracy: 13/50 -> 25/50 | Helped |
| use CNN-based scoring instead of Hamming distance (ablation) | ConceptSearch · ARC evaluation set (50 tasks) | accuracy: 25/50 -> 25/50 | No clear effect |
| use CNN-based scoring instead of Hamming distance (ablation) | ConceptSearch · ARC tasks solved by both scoring methods | efficiency improvement: 29.3% | Helped |
| use LLM-based natural-language scoring instead of CNN-based scoring (ablation) | ConceptSearch · ARC evaluation set (50 tasks) | accuracy: 25/50 -> 29/50 | Helped |
| use LLM-based natural-language scoring instead of CNN-based scoring (ablation) | ConceptSearch · ARC tasks solved by both scoring methods | efficiency improvement: 10.3% | Helped |
| use LLM-based natural-language scoring instead of Hamming distance (ablation) | ConceptSearch · ARC tasks solved by both scoring methods | efficiency improvement: 24.7 | Helped |
| combine CNN-based and LLM-based scoring (ablation) | ConceptSearch · ARC evaluation tasks | task coverage | No clear effect |
| add problem-iteration by reframing the residual task using the best program (ablation) | ConceptSearch · complex ARC tasks | task-solving success | Hurt |
Surprising findings
- The CNN-based concept scorer reached only about 40% accuracy on ConceptARC classification, yet still improved search efficiency over Hamming distance.
- No task was solved by the CNN-based scorer that the LLM-based scorer did not solve, so combining them was not shown to add solved tasks.
Limitations
- Evaluated on only 50 ARC training-set tasks, with small sample sizes for efficiency comparisons.
- CNN concept classifier is weak (~40% accuracy) and trained on only 176 ConceptARC tasks.
- Relies on hand-crafted ARC-DSL and LLM-generated hypotheses that may be inaccurate.
Experimental setup
Datasets: ARC ConceptARC LARC ARC-DSL
Benchmarks: ARC
Metrics: task accuracy (solved out of 50) mean program iterations to solution efficiency improvement (%)
Baselines: GPT-4 direct grid prompting Hamming distance scoring
Builds on: FunSearch ARC-DSL ConceptARC LARC all-mpnet-base-v2
Closest papers by meaning
- ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (Student Abstract) (AAAI 2025 · similarity 0.98)
- Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective (ICML 2025 · similarity 0.88)
- Planning in Natural Language Improves LLM Search for Code Generation (ICLR 2025 · similarity 0.87)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.87)
- CORE: Concept-Oriented Reinforcement for Bridging the Definition–Application Gap in Mathematical Reasoning (ICLR 2026 · similarity 0.86)
- HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.