Research map / Reinforcement learning / RL from feedback
Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment
Ruoxi Cheng, Hao-Xuan Ma, Weixin Wang, Ranjie Duan, Jiexi Liu, Xiaoshuang Jia, Simeng Qin, Xiaochun Cao, Yang Liu, Xiaojun Jia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic hardness scaling on top of IRL reward models (ablation) | GRPO with IRL reward (Llama-3.1-8B) · StrongReject | goodness score: 0.8917 -> 0.9361 | Improved |
| add dynamic hardness scaling on top of IRL reward models (ablation) | GRPO with IRL reward (Qwen-2-7B) · StrongReject | goodness score: 0.5960 -> 0.8798 | Improved |
| replace static reward with IRL shadow reward models (ablation) | GRPO (Llama-3.1-8B) · WildChat | refusal rate: 55.61% -> 67.54% | Improved |
| replace static reward with IRL shadow reward models (ablation) | GRPO (Qwen-2-7B) · StrongReject | goodness score: 0.5155 -> 0.5960 | Improved |
| use product of data and model hardness instead of equal-weight sum (ablation) | DR-IRL (Llama-3.1-8B) · StrongReject | goodness score: 0.9139 -> 0.9361 | Improved |
| use product instead of tuned weighted sum (ablation) | DR-IRL (Qwen-2-7B) · WildChat | refusal rate: 78.13% -> 81.53% | Improved |
| use seven per-category reward models instead of a single reward model (ablation) | DR-IRL (Llama) · StrongReject | goodness score: 0.9182 -> 0.9361 | Improved |
| use seven per-category reward models instead of a single reward model (ablation) | DR-IRL (Llama) · Stereotype | refusal rate: 96.25% -> 98.87% | Improved |
| include both data and model hardness coefficients (vs none) (ablation) | DR-IRL (Llama-3.1-8B) · StrongReject | goodness score: roughly 4 percentage points | Improved |
| replace CoT with CoD prompting for data generation (ablation) | Qwen-2-7B · CoD vs CoT accuracy | accuracy: 91.9% -> 94.1% | Improved |
| replace CoT with CoD prompting for data generation (ablation) | Llama-3.1-8B · CoD vs CoT accuracy | accuracy: 86.2% -> 87.9% | Improved |
| apply hardness coefficient to DPO | DPO (Llama-3.1-8B) · StrongReject | goodness score: 0.5054 -> 0.5826 | Improved |
Closest papers by meaning
- Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment (AAAI 2025 · similarity 0.92)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.90)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
