Research map / Vision-language and multimodal / Human and embodied modeling
Neurosymbolic Grounding for Compositional World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cosmos, an object-centric neurosymbolic world model that pairs neural slot encodings with CLIP-derived symbolic attributes and neurosymbolic attention over learned rules, evaluated for compositional generalization on the 2D block pushing domain.
Problem
Learning action-conditioned object-centric world models that compositionally generalize to unseen scenes formed from known visual attributes (entity and relational composition), which purely neural methods struggle with.
Approach
Each entity is encoded with both a neural slot vector and a set of symbolic attribute embeddings whose values are selected zero-shot by CLIP; a neurosymbolic attention matches symbolic keys against learned neural rule queries to route transitions to modular rule MLPs.
Key results
- Cosmos achieves the lowest next-state MSE on all datasets, e.g., 7.66E-04 vs 3.51E-03 for AlignedNPS on 3-object entity composition.
- On 3-object RC-Sticky, Cosmos MSE is 4.23E-03 versus 1.14E-02 for AlignedNPS and 7.94E-03 for GNN.
- Cosmos shows the most consistent downstream greedy planning deviation across datasets, per the paper's planning evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · RC-Sticky, 3 objects | MSE: 1.14E-02 -> 4.23E-03 | Helped |
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · RC-Sticky, 5 objects | MSE: 6.07E-03 -> 4.15E-03 | Helped |
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · RC-Team, 3 objects | MSE: 1.24E-02 -> 4.60E-03 | Helped |
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · RC-Team, 5 objects | MSE: 9.64E-03 -> 5.53E-03 | Helped |
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · EC, 3 objects | MSE: 3.51E-03 -> 7.66E-04 | Helped |
| Add neurosymbolic grounding and attention rather than using the neural-only AlignedNPS model (ablation) | World modeling · EC, 5 objects | MSE: 2.45E-03 -> 4.08E-04 | Helped |
| Replace the neural transition representation with only symbolic embeddings (ablation) | Cosmos · RC-Sticky, 3 objects | MSE: 4.23E-03 -> 1.36E-02 | Hurt |
| Replace the neural transition representation with only symbolic embeddings (ablation) | Cosmos · RC-Team, 3 objects | MSE: 4.60E-03 -> 1.39E-02 | Hurt |
| Replace the neural transition representation with only symbolic embeddings (ablation) | Cosmos · EC, 3 objects | MSE: 7.66E-04 -> 1.21E-02 | Hurt |
Surprising findings
- The GNN achieves the highest Eq.MRR on entity composition despite having the worst autoencoder reconstruction error, suggesting MRR is not a reliable indicator of downstream utility.
- A symbols-only transition model performs much worse (e.g., 1.21E-02 MSE on 3-object EC) than the neural model, showing symbols alone are insufficient.
Limitations
- Baseline codebases could not be reproduced for 3x224x224 images, so reimplementations were used.
- Evaluated only on a single synthetic 2D block pushing domain with 3 and 5 objects.
- Absolute Eq.MRR results are inconsistent across models, so the representation-separation claim is weak.
Experimental setup
Datasets: 2D block pushing domain (synthetic, generated)
Benchmarks: 2D block pushing domain (EC, RC-Sticky, RC-Team)
Metrics: MSE AE-MSE Equivariant MRR Hungarian distance planning error
Baselines: GNN (homomorphic world model) AlignedNPS Only Symbols ablation
Builds on: Neural Production Systems Segment Anything CLIP Homomorphic world models
Closest papers by meaning
- Bridging the gap to real-world language-grounded visual concept learning (NeurIPS 2025 · similarity 0.86)
- Dreamweaver: Learning Compositional World Models from Pixels (ICLR 2025 · similarity 0.86)
- Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models (AAAI 2026 · similarity 0.85)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.85)
- NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization (AAAI 2025 · similarity 0.85)
- Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.