Research map / Reinforcement learning / RL from feedback

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LLM-as-a-Tutor, where one LLM acts as examiner (pairwise rollout comparison to detect non-challenging prompts) and generator (appending atomic constraints plus rubric criteria), enabling policy-aware prompt adaptation for rubric-based non-verifiable RL.

Problem

Rubric-based RL for non-verifiable instruction following uses static prompt corpora, so many prompts fail to induce reward variance among rollouts, leaving the judge without a discriminative signal.

Approach

A single tutor LLM pairwise-compares two policy rollouts to detect saturated prompts and, when saturated, appends one atomic constraint and matching rubric criteria. The append-only design monotonically raises difficulty while preserving the seed distribution, so the curriculum adapts to the policy without external schedules.

Key results

• LLM-as-a-Tutor achieves average 51.96 vs. 51.04 for the best baselines (Policy-adaptive rubrics and EVA) and 49.12 for untrained Qwen3-1.7B.
• Ablation: adaptive append selection averages 43.19 vs. 42.13 for Always and 42.82 for Random; append outperforms rewrite (43.19 vs. 42.88).
• Modification ratio rises monotonically with policy scale from 8.1% (0.6B) to 40.5% (4B) with the 8B tutor fixed.

Surprising findings

• Policy-adaptive rubrics underperform on average despite adapting to the policy, suggesting some prompts cannot yield discriminative signal via rubric refinement alone.
• Evol-Instruct, which rewrites prompts without policy-awareness, underperforms even base rubrics (50.24 vs. 50.51).

Limitations

• Main results only for a single policy-tutor pair (Qwen3-1.7B with Qwen3-8B); no other model families evaluated.
• Relies on tutor capability for reliable pairwise saturation judgments and constraint generation.
• Additional tutor inference cost during training.

Experimental setup

Datasets: WildChat FollowBench AdvancedIF InfoBench

Benchmarks: FollowBench AdvancedIF InfoBench

Metrics: HSR SSR Overall Micro DRFR average score

Baselines: Distillation Base rubrics WildChecklists Policy-adaptive rubrics Evol-Instruct EVA Variance-based selection

Builds on: GRPO RLCF DR Tulu EVA Evol-Instruct

Closest papers by meaning

• Reward Is Enough: LLMs Are In-Context Reinforcement Learners (ICLR 2026 · similarity 0.89)
• Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs (ICML 2025 · similarity 0.89)
• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.89)
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
• AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking (NeurIPS 2025 · similarity 0.88)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.88)