Research map / Large language models / Agents, code and math
ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies FunSearch-style LLM program evolution to ARC with concept-based scoring functions (CNN and LLM-embedding similarity) to guide program search more efficiently than Hamming distance.
Problem
Direct LLM prompting solves few ARC tasks, and program search over ARC needs a scoring signal that captures transformation concepts rather than superficial pixel similarity.
Approach
Replace the Hamming-distance score in FunSearch with concept-based similarity: a CNN trained on ConceptARC to embed input-output transformations, and a natural-language hypothesis embedding (fine-tuned SentenceTransformer on ConceptARC/LARC) used to pick programs for the LLM prompt.
Key results
- LLM-based scoring solves 29/50 ARC training tasks versus 13/50 for GPT-4 direct grid prompting (Xu et al. 2024).
- Hamming distance and CNN-based scoring each solve 25/50 tasks; LLM-based scoring needs 2.05 mean iterations versus 3.70 for Hamming distance.
- Table 2 reports LLM-based scoring gives 24.7% efficiency gain over Hamming distance and CNN-based scoring gives 29.3% over Hamming distance on commonly solved tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-based natural-language scoring instead of Hamming-distance scoring (ablation) | ConceptSearch · ARC evaluation set (50 training tasks) | accuracy: 25/50 -> 29/50 | Helped |
| Use LLM-based natural-language scoring instead of Hamming-distance scoring (ablation) | ConceptSearch · ARC evaluation set (50 training tasks) | mean iterations: 3.70 -> 2.05 | Helped |
| Use CNN-based scoring instead of Hamming-distance scoring (ablation) | ConceptSearch · ARC evaluation set (50 training tasks) | accuracy: 25/50 -> 25/50 | No clear effect |
| Use CNN-based scoring instead of Hamming-distance scoring (ablation) | ConceptSearch · ARC evaluation set (50 training tasks) | mean iterations: 3.70 -> 2.80 | Helped |
| Use ConceptSearch with LLM-based scoring instead of direct-grid GPT-4 prompting | ARC · ARC evaluation set (50 training tasks) | accuracy: 13/50 -> 29/50 | Helped |
Surprising findings
- The CNN-based concept scorer gives no accuracy gain over Hamming distance (25/50 each) despite being concept-based.
Limitations
- Evaluated only on a 50-task subset of ARC training tasks.
- Single-run results with no variance reported.
- CNN scorer shows no accuracy improvement over Hamming distance.
Experimental setup
Datasets: ARC-DSL ConceptARC LARC ARC training tasks
Benchmarks: ARC (50 training tasks subset)
Metrics: accuracy (tasks solved) mean iterations efficiency gain (%)
Baselines: GPT-4 direct grid prompting (Xu et al. 2024) Hamming distance scoring
Builds on: FunSearch ARC-DSL ConceptARC LARC
Closest papers by meaning
- ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC) (AAAI 2025 · similarity 0.98)
- Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective (ICML 2025 · similarity 0.88)
- Planning in Natural Language Improves LLM Search for Code Generation (ICLR 2025 · similarity 0.87)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.86)
- Automating Thought of Search: A Journey Towards Soundness and Completeness (Student Abstract) (AAAI 2025 · similarity 0.86)
- CORE: Concept-Oriented Reinforcement for Bridging the Definition–Application Gap in Mathematical Reasoning (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.