Research map / Reinforcement learning / Offline RL

Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

ICLR 2024 · Reinforcement learning › Offline RL · approach: Simple practical offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes COCOA, which applies bilinear transduction with a learned anchor-seeking policy and reverse dynamics model to make offline RL policies and Q-functions conservative in a compositional input space, improving CQL, IQL, MOPO, and MOBILE on D4RL.

Problem

Offline RL suffers from distributional shift, where states and actions at execution time fall outside the training data; existing remedies are mostly behavioral conservatism, and the paper targets out-of-support generalization of function approximators in offline RL.

Approach

Reparameterize the state into an in-distribution anchor and a delta, and learn an anchor-seeking policy (trained on reverse-model rollouts and steered by the learned forward dynamics) to find anchors near the offline data, then feed both into a bilinear transductive architecture for the policy and Q-function, independent of behavioral conservatism.

Key results

• COCOA improves CQL on 10 of 12 D4RL Gym-MuJoCo tasks, raising average normalized return from 61.6 to 64.0.
• MOBILE+COCOA raises average from 70.6 to 74.8 on D4RL Gym-MuJoCo, and MOPO+COCOA from 67.2 to 70.1.
• Without anchor-seeking, the heuristic anchor variant drops CQL average to 48.4, below the 61.6 baseline.

Measured effects reported in the paper

Surprising findings

• Heuristic anchor selection (CQL+COCOA w/o anchor-seeking) lowers CQL's average return and performs worse than the unmodified baseline on most tasks.
• IQL+COCOA improves only 3 of 9 non-random tasks on D4RL Gym-MuJoCo, with average 77.3 vs 77.0 for IQL.

Limitations

• Experiments limited to control-based robotics with continuous state and action spaces; no discrete actions or image observations.
• Empirical exploration only; mechanism behind improvements is not fully understood.
• IQL gains are small and mixed across tasks.

Experimental setup

Datasets: D4RL Gym-MuJoCo (halfcheetah, hopper, walker2d) D4RL Adroit NeoRL

Benchmarks: D4RL Gym-MuJoCo D4RL Adroit NeoRL

Metrics: normalized average return over last 10 epochs

Baselines: CQL IQL MOPO MOBILE BC CQL+COCOA without anchor-seeking (heuristic anchor selection)

Builds on: bilinear transduction CQL IQL MOPO MOBILE

Closest papers by meaning

• Offline Reinforcement Learning with Reverse Model-based Imagination (NeurIPS 2021 · similarity 0.93)
• Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian (ICLR 2023 · similarity 0.92)
• A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization (ICLR 2023 · similarity 0.91)
• Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
• From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)