Research map / Reinforcement learning / Visual and representation RL
Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Policy Learning from Tutorial Books (PLfB) and the URI framework (understanding, rehearsing, introspecting) that distills a policy network from LLM-generated imaginary trajectories built from book knowledge, evaluated on Tic-Tac-Toe and Google Research Football.
Problem
Learning decision-making policies without real environment interaction, using knowledge contained in written tutorial books instead of collected trajectories.
Approach
Extract book knowledge into a code-based pseudocode database with LLMs, rehearse imaginary trajectories by LLM-instantiated code retrieved via knowledge-scope retrieval, then distill a policy with conservative Q-learning penalized by LLM-model uncertainty (CIQL).
Key results
- On Tic-Tac-Toe, URI achieves net win rates of +66%, +44%, and +52% against LLM-as-agent, LLM-RAG, and Random Policy respectively.
- On GRF, URI averages a 40.3% win rate across built-in AI levels versus 6.7% for LLM-as-agent and 16.7% for LLM-RAG.
- URI policy inference takes 0.009 s per action versus 2.84 s for LLM-as-agent, over 300x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use URI instead of LLM-as-agent | Policy learning for Tic-Tac-Toe · Tic-Tac-Toe vs LLM-as-agent | net win rate (W-L): -? -> +66% | Helped |
| Use URI instead of LLM-RAG | Policy learning for Tic-Tac-Toe · Tic-Tac-Toe vs LLM-RAG | net win rate (W-L): +18% -> +44% | Helped |
| Use URI instead of GPT-based agents | Football policy learning · GRF 11v11 Average | win rate: 16.7% ± 12.5% -> 40.3% ± 6.2% | Helped |
| Use URI rather than the rule-based policy | Football policy learning · GRF 11v11 Hard | win rate: 30% -> 32% ± 14% | Helped |
| Add rehearsal-generated imaginary data rather than train only on the pre-collected dataset (ablation) | URI · GRF | performance against Easy built-in AI | Helped |
| Add uncertainty penalties for reward and transition estimates (ablation) | URI policy learning · GRF | performance | Helped |
| Use URI rather than CQL trained on equivalent-scale rule-based real data (ablation) | GRF policy learning · GRF | performance | Helped |
| Use URI code retrieval rather than vector embedding or code summary | Knowledge retrieval in URI · GRF retrieval test set | top-15 hit rate | Helped |
| Distill knowledge into a URI policy network rather than use LLM-as-agent | GRF action selection · GRF | inference time per action: 2.84 ± 0.71 -> 0.009 ± 0.0004 | Helped |
| Distill knowledge into a URI policy network rather than use LLM-RAG | GRF action selection · GRF | inference time per action: 4.12 ±1.46 -> 0.009 ± 0.0004 | Helped |
Surprising findings
- URI's policy in the Hard GRF setting achieves a higher win rate (32%) than the rule-based AI used to generate its real states (30%).
- URI-w/o-penalty performs worse than URI-w/o-rehearsing, indicating unpenalized imaginary data can hurt policy learning.
Limitations
- Performance depends on the quality and coverage of the tutorial books covering dynamics, policy, and rewards.
- Football experiments use rule-based policy to execute fine-grained maneuvers and real initial states collected from the environment, so the pipeline is not fully book-only.
- Hallucinations in LLM generation and retrieval misses can corrupt the imaginary dataset.
Experimental setup
Datasets: RedPajama-1T football books (~90 books) Tic-Tac-Toe minimax trajectories with GPT-generated tutorials 7,500 real states from rule-based policy rollouts in GRF
Benchmarks: Tic-Tac-Toe PLfB benchmark Google Research Football 11 vs 11 (easy, medium, hard)
Metrics: win rate draw rate loss rate net win rate goal difference per match hit rate inference time
Baselines: LLM-as-agent LLM-RAG Random Policy Minimax-noise Rule-based-AI CQL-real-data URI-w/o-rehearsing URI-w/o-penalty
Builds on: Conservative Q-learning MOPO retrieval augmented generation offline reinforcement learning
Closest papers by meaning
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.88)
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.87)
- The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data (ICML 2025 · similarity 0.86)
- Text-Aware Diffusion for Policy Learning (NeurIPS 2024 · similarity 0.86)
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.86)
- ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.