Research map / Reinforcement learning / RL from feedback
RL with Learnable Textual Feedback: A Bilevel Approach
Utsav Singh, Sidhaarth Sredharan Murali, Souradip Chakraborty, Amrit Singh Bedi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add bilevel regularization (ablation) | Bi-NAC · MATH-500 (Llama-3.2-1B, feedback analysis) | accuracy (%): 41.80 -> 46.60 | Improved |
| RL-train the critic (higher level) while actor stays pre-trained (ablation) | Hier-NFT · MATH-500 (Llama-3.2-1B, feedback analysis) | accuracy (%): 35.40 -> 38.00 | Improved |
| also fine-tune the lower-level actor with the trained critic (ablation) | Hier-FT · MATH-500 (Llama-3.2-1B, feedback analysis) | feedback accuracy gain (%): 38.00 -> 41.80 | Improved |
| replace fixed feedback with learned feedback (ablation) | SCoRe · MATH-500 (Llama-3.2-1B, feedback analysis) | accuracy (%): 39.60 -> 46.60 | Improved |
| replace fixed feedback with learned feedback (ablation) | SCoRe · MATH-500 (Llama-3.2-1B, feedback analysis) | feedback actionability (FO): 2.1 -> 4.2 | Improved |
| use one model for both feedback and refinement (1B) instead of two models (2B) (ablation) | Bi-NAC · MATH-500 | accuracy (%): 46.56 -> 46.84 | No clear effect |
| use one model for both feedback and refinement (1B) instead of two models (2B) (ablation) | Bi-NAC · MBPP | accuracy (%): 66.73 -> 65.24 | No clear effect |
| extend refinement from turn 2 to turn 3 (ablation) | Bi-NAC · MATH-500 (Qwen3-1.7B, multi-turn) | accuracy (%): 70.2 -> 71.6 | Improved |
| extend refinement from turn 3 to turn 5 (ablation) | Bi-NAC · MATH-500 (Qwen3-1.7B, multi-turn) | accuracy (%): 71.6 -> 70.8 | No clear effect |
| use Bi-NAC feedback in refinement instead of untrained baseline at turn 2 | Qwen3-1.7B · MATH-500 (Qwen3-1.7B, multi-turn) | accuracy (%): 65.6 -> 70.2 | Improved |
| swap LLaMA-1B critic and actor for Qwen-1.5B (ablation) | Bi-NAC · MATH-500 (heterogeneous pairing) | accuracy (%): 44.6 -> 48.7 | Mixed |
| Bi-NAC (2B) instead of GRPO (3B) | GRPO · MATH-500 (compute trade-off) | accuracy (%): 41.43 -> 46.56 | Improved |
Closest papers by meaning
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.91)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning (NeurIPS 2026 · similarity 0.90)
- Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning (ICLR 2026 · similarity 0.90)
- Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
