Research map / Reinforcement learning / Policy gradient and actor-critic
The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distributional Reward Critic (DRC) and General DRC (GDRC), which cast reward estimation as ordinal-classification to learn under unknown generalized confusion-matrix reward perturbations for any RL algorithm.
Problem
Reinforcement learning when observed rewards are perturbed, corrupted, or noisy, with unknown perturbation structure, where existing methods assume known perturbations, clean rewards, or optimal-policy-preserving perturbations.
Approach
Turns reward regression into ordinal classification over reward intervals, trained with an ordinal cross-entropy loss that up-weights far-off misclassifications, and replaces the agent's observed rewards with the critic's predictions; GDRC ensembles critics over different discretizations and selects the number of bins n_r via the OCE increase pattern.
Key results
- Under GCM perturbations in Mujoco, DRC wins/ties the highest return in 44/48 settings (vs. 11/48 for the best baseline RE).
- Under continuous perturbations, GDRC wins/ties 10/12 cases versus RE's 7/12.
- In clean environments GDRC is comparable to PPO and RE (e.g., Hopper 2753.7 vs. RE 2689.2 and PPO 2122.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace cross-entropy with ordinal cross-entropy (ablation) | DRC · Hopper, HalfCheetah, Walker2d, and Reacher under GCM perturbations | episodic return: 2198.8 -> 2677.5; 851.5 -> 2127.3; 1747.6 -> 2058.6; -12.2 -> -7.8 | Helped |
| use a sliding window of the latest 20 epochs to estimate the reward range rather than retaining all reward samples (ablation) | GDRC · HalfCheetah clean | episodic return: 1667.9 -> 2698.0 | Helped |
| set the reward critic discretization n_o to a multiple of the perturbation interval count n_r (ablation) | DRC · Hopper under GCM perturbations | episodic return | Helped |
| increase n_o beyond n_r without sufficient samples (ablation) | DRC · reward reconstruction across simulation environments | Reconstruction Error | Hurt |
| use the latest 20 epochs rather than keeping samples for too long (ablation) | GDRC · reward distribution shifts during training | training speed | Helped |
| use DRC instead of DRC_CE (ablation) | DRC · HalfCheetah and Reacher under GCM perturbations | episodic return | Helped |
| increase the GCM perturbation ratio ω | RL methods under GCM perturbations · Mujoco environments | episodic return | Hurt |
| use GDRC rather than PPO or RE under continuous perturbations | PPO-based RL · Mujoco continuous perturbations | episodic return | Mixed |
| use GDRC rather than PPO in clean environments | PPO-based RL · Hopper, HalfCheetah, Walker2d, and Reacher | episodic return | Mixed |
Surprising findings
- SR_W, which has access to the true confusion matrix, performs worse than PPO in HalfCheetah and Walker2d.
- Mild reward noise appears to help PPO explore in Hopper, improving clean-environment performance.
Limitations
- DRC requires known GCM structure (n_r, reward range, and interval length), so it is not applicable without prior knowledge.
- GDRC performance degrades under large perturbation ratios, e.g., Pendulum at omega=0.9 where it is outperformed by DRC.
- SR requires a non-tunable rescaling hyperparameter and is only applicable to discrete-reward settings.
Experimental setup
Datasets: Hopper-v3 HalfCheetah-v3 Walker2d-v3 Reacher-v2 Pendulum-v1 CartPole-v1
Benchmarks: Hopper HalfCheetah Walker2d Reacher Pendulum CartPole
Metrics: episodic return standard error
Baselines: PPO RE SR SR_W DDPG DQN
Builds on: PPO DDPG DQN Reward Estimation Surrogate Reward
Closest papers by meaning
- Reinforcement Learning with Perturbed Rewards (AAAI 2020 · similarity 0.91)
- A Differential Perspective on Distributional Reinforcement Learning (AAAI 2026 · similarity 0.89)
- The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret (ICML 2025 · similarity 0.89)
- The Value of Reward Lookahead in Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Reinforcement Learning with Lookahead Information (NeurIPS 2024 · similarity 0.88)
- Information-Theoretic Reward Decomposition for Generalizable RLHF (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.