Research map / Reinforcement learning / Offline RL

A Tractable Inference Perspective of Offline RL

NeurIPS 2024 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Trifle, which uses tractable probabilistic circuits to exactly compute conditional/expected-return queries so sequence-model policies (TT, DT) can sample high-return actions at inference time, improving offline RL on Gym-MuJoCo and stochastic and action-constrained tasks.

Problem

Offline RL via sequence modeling (RvS) often fails at evaluation time because sampling high-return actions requires intractable conditional/constrained generation under stochastic data-collection policies and dynamics.

Approach

Use a probabilistic circuit (HCLT) to exactly compute marginal and conditional probabilities of value estimates, and use these as correction terms in an autoregressive proposal distribution from a GPT prior, followed by beam search over expected returns computed exactly by the PC.

Key results

• Trifle (TT base) achieves average normalized score 83.1 across 9 Gym-MuJoCo tasks, the highest among compared methods, and state-of-the-art on 7 of 9.
• On stochastic Taxi, m-Trifle reaches episode return -57 with failure probability 0.02, versus -182 for TT and -388 for DT.
• On action-constrained Med-Expert MuJoCo variants, Trifle outperforms TT on all three (e.g., Halfcheetah 81.9 vs 77.8, Hopper 109.6 vs 100.0).

Measured effects reported in the paper

Surprising findings

• Single-step Trifle (s-Trifle) outperforms TT in the stochastic Taxi environment despite TT estimating values more accurately, attributed to TT's inefficiency at sampling rewarding actions.
• The reported theory shows that computing the return-conditioned action distribution is NP-hard even under a Naive Bayes model, motivating approximate sampling.

Limitations

• Performance depends on the accuracy of the expressive tractable model (PC) trained on sequential data.
• Current probabilistic circuit implementations are slower than neural network libraries.

Experimental setup

Datasets: D4RL Gym-MuJoCo (HalfCheetah, Hopper, Walker2d medium/medium-replay/medium-expert) Stochastic Taxi (collected with Q-learning agent) Stochastic FrozenLake (DQN-perturbed data)

Benchmarks: Gym-MuJoCo (9 tasks) Stochastic Taxi Stochastic FrozenLake Action-space-constrained Gym-MuJoCo variants

Metrics: normalized score average score episode return inference-time optimality quantile penalty count failure probability

Baselines: Trajectory Transformer Decision Transformer TT(+Q) Decision Diffuser IQL CQL %BC TD3(+BC) Dichotomy of Control QDT

Builds on: Decision Transformer Trajectory Transformer Hidden Chow-Liu Tree probabilistic circuits Decision Diffuser IQL

Closest papers by meaning

• A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Scaling Offline RL via Efficient and Expressive Shortcut Models (NeurIPS 2025 · similarity 0.90)
• Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.90)
• On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.89)
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.89)
• Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.89)