Research map / Large language models / Reasoning and chain of thought
Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Echo-GRPO, which rewrites off-policy teacher reasoning traces into the student's own idiolect via Dual-Reference Decoding so that GRPO's importance ratios stay near unity, and instantiates it as VideoEcho-R1 for video reasoning distillation.
Problem
Mixed-policy GRPO injects off-policy privileged teacher traces that are low-probability under the student, causing trust-region clipping to suppress gradients on semantically critical reasoning tokens and degrading video reasoning.
Approach
Paraphrase privileged teacher traces into the student's native distribution using Dual-Reference Decoding, a product-of-experts combining a semantic reference (top-k of the teacher-conditioned distribution) and a distributional reference (student's unconditioned distribution), so the rewritten traces stay on-policy while preserving semantics.
Key results
- VideoEcho-R1 (Qwen3-VL-4B) reaches 57.2 average across five video benchmarks, vs 55.9 for vanilla GRPO and 39.5 for Mixed-Policy GRPO.
- On the ablation over three benchmarks, DRD reaches 58.2 average vs 52.7 for generic paraphrasing and 56.8 for semantic-reference-only.
- Adding idiolectal paraphrasing to RL w/ SFT raises the 5-benchmark average from 50.4 to 53.0.
Surprising findings
- Mixed-Policy GRPO underperforms vanilla GRPO substantially on Qwen3-VL-4B (39.5 vs 55.9 average), and collapses in early training.
- Vanilla GRPO remains competitive or superior on VSI-Bench, a numerical estimation benchmark.
Limitations
- Quality of paraphrased traces is bounded by the teacher's reasoning quality.
- Gains are reduced on tasks requiring fine-grained numerical precision such as VSI-Bench.
- Small training set (2.4K samples) in main experiments; potential benchmark contamination from pretraining.
Experimental setup
Datasets: OneThinker-SFT-340K OpenThought
Benchmarks: VideoMMMU MMVU Video-Holmes VSI-Bench VideoMME AIME24 AIME25 HMMT25
Metrics: accuracy
Baselines: GRPO SFT SFT → GRPO Mixed-Policy GRPO RL w/ SFT LUFFY OPSD-JSD OPSD-IKL
Builds on: GRPO Mixed-Policy GRPO LUFFY OPSD EasyR1
Closest papers by meaning
- Find, Fix, Reason: Context Repair for Video Reasoning (ICML 2026 · similarity 0.91)
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.90)
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.89)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Video-R1: Reinforcing Video Reasoning in MLLMs (NeurIPS 2025 · similarity 0.89)
- Learning to Reason under Off-Policy Guidance (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.