Research map / Multi-agent systems and games / Human-AI and goal-conditioned
VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Neuro-Symbolic Predicates, VLM-invented Python predicates learned online with a new precondition learner, used for abstract world-model planning in five simulated robot domains.
Problem
Robots need abstractions that are both perceptually and logically rich to plan in novel environments, but hand-coded symbolic world models cannot adapt and prior predicate learners are restricted in form and need demonstrations.
Approach
Neuro-Symbolic Predicates are Python code snippets that call VLMs for visual queries and combine them with programming constructs; VLMs propose predicates, which are selected by a classification-accuracy objective and used to learn HLA preconditions optimistically from online exploration without demonstrations.
Key results
- Ours consistently outperforms MAPLE and ViLa across all five domains, achieving near-perfect solve rates on out-of-distribution test tasks.
- Ours achieves similar solve rates and planning efficiency to the Oracle with manually designed predicates and operators.
- Sym. pred. fails on perceptually grounded predicates such as jug-in-machine (Coffee) and heavy blocks (Cover Heavy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VisualPredicator instead of MAPLE | VisualPredicator · five robotic domains, training-distribution tasks | solve rate | Helped |
| use VisualPredicator instead of ViLa | VisualPredicator · five robotic domains | solve rate | Helped |
| learn predicate abstractions rather than use the initial underspecified representations | VisualPredicator · five robotic domains | solve rate | Helped |
| use learned abstractions instead of manually designed abstractions | VisualPredicator · five robotic domains | solve rate | No clear effect |
| use neuro-symbolic predicates with perceptual queries and derived predicates instead of feature-based symbolic predicates | VisualPredicator · Coffee | solve rate | Helped |
| use neuro-symbolic predicates with perceptual queries and derived predicates instead of feature-based symbolic predicates | VisualPredicator · Cover Heavy | solve rate | Helped |
| use neuro-symbolic predicates with derived predicates instead of feature-based symbolic predicates | VisualPredicator · Balance | solve rate | Helped |
| use the extended operator learner instead of the unmodified operator learner (ablation) | VisualPredicator · five robotic domains | solve rate | Helped |
| use the extended operator learner instead of the unmodified operator learner (ablation) | VisualPredicator · held-out tasks with more cups | planning efficiency | Helped |
Surprising findings
- Initial VLM-based predicate proposals about material (e.g., IsWood, IsMetal) were not interpreted reliably by the evaluation VLM, and the domain was only resolved by changing block colors.
- ViLa frequently produces infeasible plans, such as picking blocks from a stack's middle or grasping a jug without considering orientation.
Limitations
- Accuracy and reliability of the VLM in evaluating predicates is the primary bottleneck.
- Evaluated only in simulated domains with poor graphics quality; no partial observability or real-world tests.
- Assumes pre-defined motor skills and object-centric state; no recovery mechanisms for failed plans.
Experimental setup
Datasets: online interaction data in simulated PyBullet environments
Benchmarks: Cover Blocks Coffee Cover Heavy Balance
Metrics: percentage of test tasks solved planning budget used
Baselines: MAPLE ViLa Sym. pred. No invent Oracle Ablate op.
Builds on: cluster and intersect operator learning ViperGPT VisProg MAPLE ViLa
Closest papers by meaning
- Learning Neuro-Symbolic Abstractions for Robot Planning and Learning (AAAI 2024 · similarity 0.89)
- Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.89)
- NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domains (ICLR 2025 · similarity 0.88)
- R2C: Mapping Room to Chessboard to Unlock LLM As Low-Level Action Planner (CVPR 2025 · similarity 0.88)
- Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.88)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.