Research map / Reinforcement learning / Visual and representation RL
Neurosymbolic World Models for Sequential Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWMPO, an unsupervised method that learns neural world-model primitives and synthesizes environment-specific neurosymbolic finite state machine world models for model-based RL on low-level non-visual continuous observations.
Problem
Traditional world models are unstructured neural networks that do not explicitly represent high-level modes (e.g., water vs. land) of an environment, which hinders reuse across environments and sample-efficient policy optimization on low-level continuous observations.
Approach
Learn modular neural world-model primitives unsupervised from offline data via a first-order model with a learned mode-embedding (mutual-information regularized, clustered with k-means), then assemble them into an FSM for a new environment using decision-tree transition predicates, and use it in a model-based SAC loop.
Key results
- On mode-tracking Levenshtein error, SWMPO outperforms HMM, SLDS, and rSLDS baselines in Point Mass and Salamander, and significantly beats SLDS/rSLDS while only marginally beating HMM in LiDAR Racing.
- In Point Mass model-based RL (64 seeds), SWMPO outperforms both model-free SAC and a model-based baseline with an online neural forward model.
- In BipedalWH, SWMPO only marginally beats baselines and all models struggle to capture the environment's structure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SWMPO instead of the evaluated baselines | Point Mass · Point Mass | Levenshtein distance | Helped |
| Use SWMPO instead of the evaluated baselines | Salamander · Salamander | Levenshtein distance | Helped |
| Use SWMPO instead of SLDS and rSLDS baselines | LiDAR Racing · LiDAR Racing | Levenshtein distance | Helped |
| Use SWMPO instead of the HMM baseline | LiDAR Racing · LiDAR Racing | Levenshtein distance | Helped |
| Use SWMPO instead of the evaluated baselines | Bipedal Walker · Bipedal Walker | Levenshtein distance | Helped |
| Use SWMPO model-based RL instead of model-free RL | Point Mass · Point Mass | reward | Helped |
| Use SWMPO model-based RL instead of an online monolithic neural model | Point Mass · Point Mass | reward | Helped |
Surprising findings
- In BipedalWH all models, including SWMPO, struggle to capture the structure and dynamics of the environment.
Limitations
- Assumes the number of modes is known in advance.
- Assumes active POMDP modes are a subset of offline POMDP modes (global modes assumption).
- Limited to low-level, non-visual continuous observations; high-dimensional visual domains not addressed.
Experimental setup
Datasets: Point Mass simulated trajectories LiDAR Racing trajectories Salamander Webots trajectories Bipedal Walker trajectories
Benchmarks: Point Mass LiDAR Racing Salamander Bipedal Walker
Metrics: Levenshtein distance between predicted and ground-truth mode episode reward
Baselines: HMM SLDS rSLDS model-free SAC model-based RL with online neural forward model
Builds on: Soft Actor-Critic Hidden Markov models Switching linear dynamical systems recurrent switching linear dynamical systems Finite state machine policies (Inala et al., 2020)
Closest papers by meaning
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.87)
- Neural Motion Simulator Pushing the Limit of World Models in Reinforcement Learning (CVPR 2025 · similarity 0.86)
- WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.86)
- HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planning (NeurIPS 2026 · similarity 0.86)
- Simplifying Latent Dynamics with Softly State-Invariant World Models (NeurIPS 2024 · similarity 0.85)
- Do Transformer World Models Give Better Policy Gradients? (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.