Research map / Large language models / Instruction and fine-tuning
Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies large-scale GRPO with rule-based DRG rewards on top of SFT (Qwen2.5-7B-Instruct) to build DRG-Sapphire for MS-DRG coding from MIMIC-IV discharge summaries, and studies how SFT vs RL data allocation and GRPO enhancements affect OOD reasoning.
Problem
LLMs perform poorly on MS-DRG coding from clinical notes because the task is out-of-distribution for pretraining corpora, and it is unclear how RL should be applied to such knowledge-intensive OOD tasks.
Approach
Combines a CoT cold-start SFT on LLM-generated reasoning with GRPO using tiered DRG rule-based accuracy rewards, and tests a set of GRPO modifications (dynamic resampling, DAPO/Dr.GRPO losses, KL decay, reward shaping, curriculum and staged learning), finding RL gains scale log-linearly with SFT size.
Key results
- DRG-Sapphire reaches 54.8% DRG Pass@1 on MIMIC-IV test set, versus 53.9% for DRG-LLaMA-7B.
- RL performance scales approximately linearly with the logarithm of the number of SFT examples.
- Across SFT-GRPO splits on DRG-Small, GRPO improves Pass@1 by about 10 points but reduces Pass@8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRG-Sapphire instead of DRG-LLaMA-7B | DRG coding · MIMIC-IV test | accuracy: 53.9% -> 54.8% | Helped |
| Add positive-reward-constrained dynamic resampling (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 39.2 | Helped |
| Use neutral dynamic resampling (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 20.3 | Hurt |
| Encourage CoT-First reasoning (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 35.5 | Hurt |
| Encourage Differential Thinking (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 30.2 | Hurt |
| Use DAPO loss (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 40.1 | Helped |
| Use Dr. GRPO loss (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 37.5 | Hurt |
| Remove the KL penalty (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 39.8 | Helped |
| Apply cosine KL decay (ablation) | Best GRPO configuration · MIMIC-IV test, 90% SFT | Pass@1: 54.2 -> 54.6 | Helped |
| Use strict rather than dense accuracy reward (ablation) | Vanilla GRPO · DRG-Small | Pass@1: 38.5 -> 40.1 | Helped |
| Use balanced rather than dense accuracy reward (ablation) | Vanilla GRPO · DRG-Small | Pass@1: 38.5 -> 38.1 | Hurt |
| Exclude hard training cases (ablation) | Vanilla GRPO + Dense Reward · DRG-Small | Pass@1: 38.5 -> 40.4 | Helped |
Surprising findings
- Removing the KL penalty frequently caused training collapse, contrary to findings in math RL.
- Answer-first reasoning emerged naturally and outperformed forced CoT-first and differential-thinking variants.
Limitations
- Only rule-based outcome rewards; no process supervision over reasoning steps.
- Curriculum and case filtering were static, computed once from a base run.
- Physician reviewers found occasional factual errors in reasoning; evaluated on a single task (DRG coding).
Experimental setup
Datasets: MIMIC-IV (DRG-LLaMA train/test split) DRG-Small subset
Benchmarks: MIMIC-IV DRG test set (N=26,244) internal institutional dataset (2,500 cases)
Metrics: Pass@1 Pass@8 Maj@8 physician rating of helpfulness and accuracy
Baselines: DRG-LLaMA-7B DeepSeek-R1-distilled Qwen 32B proprietary reasoning models non-reasoning LLMs vanilla GRPO
Builds on: GRPO DeepSeek-R1 DAPO Dr. GRPO DRG-LLaMA
Closest papers by meaning
- MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.86)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.86)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.86)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.86)
- d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.