Research map / Reinforcement learning / RL from feedback
Text2Grad: Reinforcement Learning from Natural Language Feedback
Hanyang Wang, Lu Wang, Chaoyun Zhang, Tianjun Mao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| span-level NL-feedback gradients (Text2Grad) instead of scalar-reward PPO | Llama-3.1-8B-Instruct · SLF5K | ROUGE-L: 0.262 -> 0.291 | Improved |
| Text2Grad instead of PPO on code | Llama-3.1-8B-Instruct · KodCode (avg pass@1) | pass@1: 62.5 -> 66.1 | Improved |
| Text2Grad instead of PPO on QA | Llama3-8B-Instruct · UltraFeedback AlpacaEval 2.0 | win rate: 32.4 -> 34.7 | Improved |
| Text2Grad instead of PPO/DPO on Mistral-7B | Mistral-7B-Instruct-v0.2 · MT-Bench | score: PPO 6.55 -> 6.78; base 6.60 | Mixed |
| add CoT reasoning before span annotation (ablation) | Text2Grad · SLF5K | ROUGE-L: 0.275 -> 0.291 | Improved |
| add CoT reasoning before span annotation (ablation) | Text2Grad · KodCode (avg pass@1) | pass@1: 59.2 -> 66.1 | Improved |
| add CoT reasoning before span annotation (ablation) | Text2Grad · UltraFeedback AlpacaEval 2.0 | win rate: 28.6 -> 34.7 | Improved |
| use span-level labels instead of dense token-level labels (ablation) | Text2Grad · SLF5K | ROUGE-L: 0.196 -> 0.291 | Improved |
| add CoT to reward model annotation (span precision) (ablation) | reward model · SLF5K positive-token precision/recall | precision/recall: precision 0.63 -> 0.58; recall 0.46 -> 0.63 | Mixed |
| add CoT to reward model (ablation) | reward model · UltraFeedback positive-token recall | recall: 0.59 -> 0.43 | Hurt |
| Text2Grad on code vs ablated no CoT on ARC-Challenge (ablation) | Text2Grad · UltraFeedback ARC-C | accuracy: 83.1 -> 84.4 | Improved |
Closest papers by meaning
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.91)
- Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.90)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
- Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization (ICML 2025 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
