Research map / Reinforcement learning / Visual and representation RL

Reinforcement Learning with Adaptive Reward Modeling for Expensive-to-Evaluate Systems

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaReMo, which decouples slow reward evaluation from fast RL decisions by training localized neural reward models that are continually fine-tuned on the agent's latest explored samples, giving large speedups on three expensive-reward tasks.

Problem

RL training is prohibitively slow when reward evaluation is computationally expensive (seconds per sample), and fixed proxy or world-model rewards become inaccurate as the agent's policy and explored state distribution shift.

Approach

Decouple online policy optimization from offline reward evaluation; a neural reward model is repeatedly fine-tuned on recent agent-explored samples labeled by direct evaluation, so each model approximates the reward only within the currently explored localized subspace, with synchronous correction, model warm-up and parallel evaluation.

Key results

• AdaReMo improves Top 5% score and Hit Ratio over the best baseline in molecular generation (e.g., PARP1 HR 0.42 vs 0.35 for FREED), with reported average gains around 13-14% over other RL methods.
• Reward model evaluation takes ~0.02s versus 8.6-15.2s for precise evaluation, described as a roughly 1,000x speedup.
• In pandemic control AdaReMo reaches Healthy 39.8 on CA-GrQc vs 35.8 for RLGN; in spatial planning travel distance drops from 3.06 to 2.78 (-9.2%).

Measured effects reported in the paper

Surprising findings

• Fixed-iteration training with the reward model is initially slower than direct evaluation (FREED), but AdaReMo overtakes it after a while.
• Setting fine-tune interval or epochs suboptimally mainly slows convergence; final performance remains within about 4% of the best setting.

Limitations

• Evaluated only with a GNN agent; generality to other architectures is left for future work.
• Reward model accuracy degrades for states outside the currently fine-tuned subspace, and hyperparameters (fine-tune interval, epochs) affect convergence speed.
• Requires periodic direct reward evaluations and a correction term from simplified evaluation, adding cost and dependence on simulator quality.

Experimental setup

Datasets: CA-GrQc SNAP real-world urban community layouts (HLG, DHM, HZ) protein targets FA7, PARP1, 5HT1B

Benchmarks: molecular generation on FA7, PARP1, 5HT1B pandemic control on CA-GrQc and SNAP urban spatial planning on HLG, DHM, HZ

Metrics: Top 5% score hit ratio Healthy Contained Distance Greenness training efficiency speedup

Baselines: HierVAE LIMO MolDQN FREED MBPO GRPO RLOO HSB KED GBP RLGN GA

Builds on: PPO RLHF MBPO FREED SAND

Closest papers by meaning

• Behavior Alignment via Reward Function Optimization (NeurIPS 2023 · similarity 0.87)
• To the Max: Reinventing Reward in Reinforcement Learning (ICML 2024 · similarity 0.87)
• R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.86)
• ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.86)
• REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.86)
• Deep Reinforcement Learning from Hierarchical Preference Design (ICML 2025 · similarity 0.86)