Research map / Reinforcement learning / Policy gradient and actor-critic
Switching the Loss Reduces the Cost in Batch Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes fitted Q-iteration with log-loss (FQI-LOG) for batch RL and proves a small-cost bound scaling with the optimal accumulated cost, validated on tabular-style control tasks with logit models and on Atari (Asterix, Seaquest) against DQN and C51.
Problem
Batch RL sample efficiency typically scales with 1/sqrt(n) regardless of the optimal cost; the question is whether FQI can achieve bounds that scale with the optimal achievable cost (small-cost bounds) so that near-zero-cost problems need fewer samples.
Approach
Replace the squared loss in FQI with log-loss, since log-loss penalizes predictions near the boundary of [0,1] differently and yields a small-cost bound via a Hellinger-distance contraction of the Bellman optimality operator, while squared loss provably cannot.
Key results
- On mountain car, FQI-LOG accumulates much smaller cost than FQI with squared loss and learns a near-optimal policy from a single successful trajectory.
- On inverted pendulum with logit models, FQI-LOG reaches good balancing policies with fewer samples than squared-loss FQI.
- On Seaquest, DQN-LOG outperforms DQN and matches C51; on Asterix, DQN-LOG performs similarly to DQN and both are below C51.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from squared loss to log-loss (FQI-LOG) | FQI · batch RL with small optimal cost | sample complexity | Helped |
| switch from squared loss to log-loss | FQI · Mountain Car | policy cost | Helped |
| switch from squared loss to log-loss | FQI · Inverted Pendulum | balance success rate | Helped |
| switch from squared loss to log-loss | DQN · Seaquest | undiscounted return | Helped |
| switch from squared loss to log-loss | DQN · Asterix | undiscounted return | No clear effect |
| clip DQNsq targets to [v_min, v_max] (ablation) | DQNsq · Atari experiments | performance | Helped |
| use squared loss instead of log-loss | FQI · batch RL with small optimal cost | sample efficiency | Hurt |
Surprising findings
- Switching only the loss from squared to log-loss in deep FQI does not consistently reproduce C51's gains; results are inconclusive across the two Atari games.
- Clipping targets of the squared-loss DQN baseline was found to improve performance, yielding a stronger baseline.
Limitations
- Theory is stated for finite function classes; extension to infinite classes such as logit models is only sketched.
- Analysis requires realizability, completeness, and finite concentrability, and assumes the function class values lie in [0,1].
- Deep RL results are inconclusive relative to C51.
Experimental setup
Datasets: Mountain car (episodic sparse cost variant) Inverted pendulum Asterix (Atari, DQN replay batches) Seaquest (Atari, DQN replay batches)
Benchmarks: Mountain car Inverted pendulum Asterix Seaquest
Metrics: accumulated cost / policy value fraction of episodes balanced for 3000 steps undiscounted return
Baselines: FQI with squared loss DQN C51
Builds on: fitted Q-iteration C51 small-loss bounds in contextual bandits (Foster et al.) distributional RL small-cost bounds (Ayoub et al.)
Closest papers by meaning
- Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL (ICML 2021 · similarity 0.90)
- Sample-Efficient Reinforcement Learning with loglog(T) Switching Cost (ICML 2022 · similarity 0.90)
- Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation (NeurIPS 2024 · similarity 0.88)
- Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching (NeurIPS 2026 · similarity 0.87)
- Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.