Research map / Reinforcement learning / RL from feedback
MaxIM: Maximally Informative Incremental Summarization via Reinforcement Learning
Jihwan Jeong, Guy Tennenholtz, Yinlam Chow, Chih-wei Hsu, Craig Boutilier
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaxIM, an RL framework that trains incremental summarization memory for bounded-memory agents by addressing sparse, delayed reward credit assignment.
Problem
Maintaining bounded memory over unbounded interaction histories, where optimizing memory representations and updates for downstream utility via RL is hampered by sparse, delayed rewards and credit assignment bottlenecks.
Approach
Formulates incremental summarization as a sequential decision problem and trains the summarizer with RL, aiming to make each memory update maximally informative for downstream task utility rather than relying on end-of-episode sparse rewards alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MaxIM information-theoretic reward shaping to RL | Task-Only RL summarizer · arXiv Abstain-MCQ (K=500) | accuracy (%): 54.44 -> 66.39 | Improved |
| add recursive consistency reward to sufficiency reward (ablation) | MaxIM · arXiv Abstain-MCQ (K=500) | accuracy (%): 51.54 -> 66.39 | Improved |
| add recursive consistency reward to sufficiency reward (ablation) | MaxIM · WildChat grounding | contradiction (%): 6.48 -> 4.27 | Improved |
| add task reward to MI shaping (ablation) | MI-Only RL · arXiv Abstain-MCQ (K=500) | accuracy (%): 50.30 -> 66.39 | Improved |
| cosine-anneal the shaping weight mid-training (45-60%) vs none (ablation) | MaxIM · arXiv Abstain-MCQ test (K=500) | accuracy (%): 57.10 -> 66.39 | Improved |
| decay shaping early (13-60%) or late (65-90%) instead of mid (ablation) | MaxIM · arXiv Abstain-MCQ test (K=500) | accuracy (%): 66.39 -> 61.18 (early), 61.12 (late) | Hurt |
| use constant MI shaping without annealing vs task-only reward (ablation) | Task-Only RL · arXiv Abstain-MCQ test (K=500) | accuracy (%): 54.44 -> 57.10 | Improved |
| train summarizer with sparse-reward RL (Task-Only) instead of SFT/prompting | Gemini 2.5 Flash summarizer · arXiv Abstain-MCQ (K=500) | accuracy (%): 31.30 -> 54.44 | Improved |
| select single-step updates by highest immediate PMI (Best-of-N MI SFT) | Policy SFT · arXiv Abstain-MCQ (K=500) | accuracy (%): 31.30 -> 29.05 | Hurt |
| use task-only or MemAgent RL summaries instead of no belief tracking | WildChat assistant · WildChat (all T) | Satisfaction delta vs No-Belief: -0.30 (Task-Only), -0.28 (MemAgent) | Hurt |
| add MaxIM shaping on WildChat T>=10 dialogues | Task-Only RL · WildChat (T>=10) | Satisfaction delta vs No-Belief: -0.37 -> +0.11 | Improved |
| increase token budget K from 100 to 800 (ablation) | MaxIM · arXiv Abstain-MCQ | accuracy (%): 34.26 -> 67.04 | Improved |
Closest papers by meaning
- Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents (NeurIPS 2026 · similarity 0.87)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.85)
- MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization (ICLR 2025 · similarity 0.85)
- Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.85)
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.85)
- S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
