Research map / Large language models / Agents, code and math
NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeSyCoCo, a neuro-symbolic visual reasoning framework that uses an LLM to generate logical programs (augmented with dependency parses), GloVe-embedded shared predicate scorers, and soft normalized composition operators.
Problem
Neuro-symbolic vision-language reasoning struggles with compositional generalization due to reliance on predefined predicates, difficulty extracting predicates from raw data, and non-differentiable composition operators.
Approach
Augments LLM program generation with dependency parses, represents each predicate by a frozen word embedding fed through a shared FFN to produce sigmoid-normalized scores, and composes them with differentiable soft operators (max/min-free conjunction via product, linear-normalized iota) instead of softmax and min.
Key results
- ReaSCAN average grounding accuracy of 97.5% versus 85.2% for Syntax Guided Transformer and 78.5% for GroCoT.
- CLEVR-CoGenT Split B accuracy of 78.8% versus 76.2% for LEFT and MDETR.
- CLEVR-SYN hard split accuracy of 73.4% versus 49.5% for LEFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeSyCoCo instead of Syntax Guided Transformer | ReaSCAN grounding · ReaSCAN average | accuracy: 85.2 -> 97.5 | Helped |
| use NeSyCoCo instead of MDETR | CLEVR-CoGenT · CLEVR-CoGenT Split B | accuracy: 76.2 -> 78.8 | Helped |
| use NeSyCoCo instead of MDETR | CLEVR-CoGenT · CLEVR-CoGenT Split A | accuracy: 99.7 -> 99.6 | No clear effect |
| use NeSyCoCo instead of LEFT | CLEVR-SYN · CLEVR-SYN Easy | accuracy: 81.9 -> 92.1 | Helped |
| add predicate embeddings (ablation) | NeSyCoCo · ReaSCAN average | accuracy: 96.5 -> 97.5 | Helped |
| add soft symbolic reasoner (ablation) | NeSyCoCo without embeddings · ReaSCAN average | accuracy: 92.5 -> 96.5 | Helped |
| add dependency parsing context to program generation (ablation) | LEFT with improved prompting · ReaSCAN average | accuracy: 92.3 -> 92.5 | Helped |
| replace GloVe with SpaCy encoder (ablation) | NeSyCoCo · CLEVR-SYN Easy | accuracy: 92.1\% -> 80.7\% | Hurt |
| replace GloVe with RoBERTa encoder (ablation) | NeSyCoCo · CLEVR-SYN Easy | accuracy: 92.1\% -> 84.7\% | Hurt |
| replace GloVe with one-hot encoding (ablation) | NeSyCoCo · CLEVR-SYN Easy | accuracy | Mixed |
Surprising findings
- LEFT's raw unbounded logit scores vary widely in scale across concepts, causing min-based composition to undervalue some predicates.
- Softmax in LEFT's iota function can inflate small score differences between similar objects.
Limitations
- Struggles with size-related concepts in ReaSCAN because size is relative and handled with unary object-level functions.
- Performance on synonyms depends on quality of frozen word embeddings (e.g., ball vs sphere, brown vs chocolate fail).
- Reasoning depends on LLM-generated programs that may contain unresolved semantic errors.
Experimental setup
Datasets: CLEVR CLEVR-CoGenT ReaSCAN CLEVR-Ref CLEVR-Puzzle CLEVR-RPM
Benchmarks: ReaSCAN CLEVR-CoGenT CLEVR CLEVR-Ref CLEVR-Puzzle CLEVR-RPM CLEVR-SYN
Metrics: accuracy object localization accuracy Pearson correlation
Baselines: LEFT MDETR GroCoT Syntax Guided Transformer NSCL VisProg ViperGPT LLaVA NeXT OpenFlamingo
Builds on: LEFT VisProg ViperGPT Neuro-Symbolic Concept Learner
Closest papers by meaning
- NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language (ICLR 2026 · similarity 0.90)
- Neural-Symbolic Recursive Machine for Systematic Generalization (ICLR 2024 · similarity 0.88)
- Concise and Logically Consistent Conformal Sets for Neuro-Symbolic Concept-Based Models (NeurIPS 2026 · similarity 0.88)
- Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.86)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.86)
- Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.