Research map / Reinforcement learning / Visual and representation RL
Learning Finite State Representations of Recurrent Policy Networks
Anurag Koul, Alan Fern, Sam Greydanus
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert quantized bottlenecks for recurrent state and observations, then fine-tune when needed | Atari RNN policies · Pong, Freeway, Bowling, Boxing | score: Pong 21 -> 21; Freeway 21 -> 21; Bowling 60 -> 60; Boxing 100 -> 100 | No clear effect |
| Insert quantized bottlenecks and fine-tune | Atari RNN policies · Breakout | score: 773 -> 423 | Hurt |
| Insert quantized bottlenecks and fine-tune | Atari RNN policies · Space Invaders | score: 1820 -> 1335 | Hurt |
| Fine-tune the inserted bottlenecks (ablation) | Pong RNN · Pong | score: 20 -> 21 | Improved |
| Fine-tune the inserted bottlenecks (ablation) | Breakout RNN · Breakout | score: 32 -> 423 | Improved |
| Fine-tune the inserted bottlenecks (ablation) | Space Invaders RNN · Space Invaders | score: 520 -> 1335 | Improved |
| Fine-tune the inserted bottlenecks (ablation) | Boxing RNN · Boxing | score: 94 -> 100 | Improved |
| Increase hidden-state bottleneck size from 8 to 16 (ablation) | Tomita Grammar 5 MMN · Tomita Grammar 5 | accuracy: 95 -> 100 | Improved |
| Fine-tune the inserted hidden-state bottleneck (ablation) | Tomita Grammar 5 MMN with B_h=8 · Tomita Grammar 5 | accuracy: 95 -> 96 | Improved |
| Use B_h=16 instead of B_h=8 (ablation) | Blind MCE MMN with B_f=8 · Blind | accuracy: 0.78 -> 1 | Improved |
| Use both observation and hidden-state bottlenecks at B_f=4 and B_h=4 (ablation) | Tracker MCE MMN · Tracker | accuracy: 1 -> 0.98 | Hurt |
| Increase the observation bottleneck size from B_f=100 to B_f=400 (ablation) | Breakout MMN with B_h=64 · Breakout | score: 423 -> 415 | No clear effect |
Closest papers by meaning
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.87)
- Recurrent Reinforcement Learning with Memoroids (NeurIPS 2024 · similarity 0.87)
- Real-Time Recurrent Learning using Trace Units in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.86)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.86)
- Rethinking Model-based, Policy-based, and Value-based Reinforcement Learning via the Lens of Representation Complexity (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
