Research map / Reinforcement learning / MDPs and planning

Learning POMDP World Models from Observations with Language-Model Priors

NeurIPS 2026 · Reinforcement learning › MDPs and planning · approach: POMDP policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pinductor uses an LLM to propose and iteratively refine executable POMDP models (transition, observation, reward, initial state) scored by a particle-filter kernel likelihood on observation-action trajectories alone, matching privileged-state LLM baselines on MiniGrid.

Problem

Learning POMDP world models from observation-action-reward trajectories without access to hidden states, which is sample-intensive for classical tabular or latent-model methods.

Approach

Score LLM-generated executable POMDP programs by a particle-filtered expected log soft-observation likelihood (distance kernel over MiniGrid observations), refine them with structured execution feedback, query-by-committee disagreement, and UCB1 tree search, without ever comparing predicted states to ground-truth states.

Key results

• Pinductor matches the average episode reward of the privileged-state POMDP Coder baseline across five MiniGrid tasks while using only observation-action-reward trajectories.
• Pinductor substantially outperforms the tabular POMDP baseline, which is granted privileged hidden-state access, in sample efficiency with few offline trajectories.
• Belief posterior mass on the true hidden state increases over episode steps, indicating useful latent-state inference during planning.

Surprising findings

• The tabular baseline with privileged hidden-state access performs worse than Pinductor despite the extra supervision.

Limitations

• Evaluation restricted to MiniGrid environments.
• High variance from reliance on LLM API calls.
• Performance depends heavily on the LLM and on availability of textual environment descriptions; degrades when semantic information is withheld.

Experimental setup

Datasets: MiniGrid offline trajectories (10 per task, mixed success and failure)

Benchmarks: MiniGrid Corners MiniGrid Empty MiniGrid Lava MiniGrid FourRooms MiniGrid Unlock

Metrics: average episode reward win rate belief entropy belief accuracy

Baselines: POMDP Coder tabular POMDP baseline random policy ground-truth models

Builds on: POMDP Coder WorldCoder particle filtering POMCP MiniGrid

Closest papers by meaning

• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.88)
• Sample-Efficient Learning of POMDPs with Multiple Observations In Hindsight (ICLR 2024 · similarity 0.88)
• The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models (ICLR 2024 · similarity 0.88)
• Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability (NeurIPS 2025 · similarity 0.87)
• WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment (NeurIPS 2024 · similarity 0.87)
• Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.87)