Research map / Reinforcement learning / RL from feedback

Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem

ICML 2024 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies forgetting of pre-trained capabilities (FPC) as a major obstacle to fine-tuning pre-trained RL agents and shows that standard knowledge retention methods (EWC, behavioral cloning, kickstarting, episodic memory) mitigate it, reaching 10K points on NetHack Human Monk.

Problem

Fine-tuning pre-trained reinforcement learning agents transfers poorly because the agent forgets pre-trained behavior on state regions not visited early in fine-tuning.

Approach

Conceptualizes poor RL transfer as forgetting of pre-trained capabilities, identifies two instances (state coverage gap and imperfect cloning gap), and treats knowledge retention techniques as the remedy for fine-tuning.

Key results

• Fine-tuning with knowledge retention in NetHack Human Monk improves the score from about 5K to over 10K points, a roughly 2x state-of-the-art improvement for neural models.
• On Montezuma's Revenge, behavioral cloning keeps Room 7 success rate stable while vanilla fine-tuning's success rate drops considerably.
• On RoboticSequence, BC solves all four stages in 80% of runs while vanilla fine-tuning and training from scratch are virtually indistinguishable.

Measured effects reported in the paper

Surprising findings

• Vanilla fine-tuning of a pre-trained RL policy can perform worse than the pre-trained model as it forgets pre-trained skills, even in stationary downstream tasks.
• Kickstarting fails completely in the state coverage gap setting because it distills on states absent from pre-training.

Limitations

• Only simple knowledge retention methods were tested; more sophisticated continual learning methods were not explored.
• Knowledge retention can hurt if the pre-trained policy is suboptimal, since it blocks improvement.
• Only models under 1B parameters were studied and only two transfer scenarios were examined.

Experimental setup

Datasets: NetHack Learning Dataset (NLD-AA) Atari Montezuma's Revenge trajectories Meta-World

Benchmarks: NetHack Learning Environment (Human Monk) Montezuma's Revenge RoboticSequence (Meta-World)

Metrics: in-game score success rate per room/stage average return

Baselines: vanilla fine-tuning training from scratch AutoAscend-distilled neural policy (prior NetHack SOTA)

Builds on: Elastic Weight Consolidation kickstarting Asynchronous PPO Soft Actor-Critic Random Network Distillation

Closest papers by meaning

• Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.89)
• Understanding Catastrophic Forgetting in Language Models via Implicit Inference (ICLR 2024 · similarity 0.88)
• Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.87)
• Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.87)
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.86)
• A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.86)