Research map / Reinforcement learning / Policy gradient and actor-critic

The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LEAST, a lightweight rule that stops and resets episodes early when the current Q-value falls below a median-based threshold scaled by gradient-based learning potential, to counter the sunk cost fallacy in off-policy deep RL.

Problem

Off-policy deep RL agents are forced to complete episodes even in unproductive trajectories, which wastes environment interactions and fills the replay buffer with uninformative transitions, hurting sample efficiency.

Approach

Learn when to stop: compare the current Q-value against a per-step median of recent episodes' Q-values, modulated by a gradient-magnitude-based learning potential weight, and terminate and reset the episode when the Q-value falls below that threshold; add entropy-aware buffer sizing and noise escalation on frequent early stops.

Key results

• On MuJoCo, LEAST improves TD3, SAC and REDQ learning efficiency and final performance across four challenging tasks, with SAC+LEAST reaching performance comparable to DroQ and REDQ+LEAST achieving the highest overall normalized scores.
• On DMC image tasks, LEAST on DrQv2 converges about 30% faster on Finger Turn Hard and outperforms CURL and A-LIX, approaching TACO without extra networks.
• Ablation: median threshold and median-based weighting are more stable than the mean; sigma-bar in [0.25, 0.35] is optimal for TD3 and [0.15, 0.25] for SAC.

Measured effects reported in the paper

Surprising findings

• LEAST may increase score variance across seeds, with wider spreads and outliers in box plots.
• Increasing UTD slowed policy optimization for TD3 with LEAST, attributed to inaccurate Q-function fitting at high UTD.

Limitations

• Stability and score variance can worsen; more robust evaluators are needed.
• Many sensitive hyperparameters (thresholds, start time, buffer size, noise schedule) are tuned with default baseline settings rather than full sweeps.
• Post-reset agents may re-enter previously suboptimal trajectories without additional noise diversification.

Experimental setup

Datasets: MuJoCo DeepMind Control Suite PointMaze

Benchmarks: MuJoCo DeepMind Control Suite PointMaze

Metrics: normalized score sample efficiency (steps to reach baseline max score) final performance fraction of active units

Baselines: TD3 SAC REDQ DroQ DrQv2 CURL A-LIX TACO

Builds on: TD3 SAC REDQ DrQv2 DroQ

Closest papers by meaning

• Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound (NeurIPS 2025 · similarity 0.90)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
• A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.87)
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.86)
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.86)
• Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.86)