Research map / Reinforcement learning / Policy gradient and actor-critic
Action abstractions for amortized sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActionPiece, which iteratively chunks frequent high-reward action subsequences (via BPE-style tokenization) into new actions added to the action space of GFlowNet, A2C, and SAC samplers, improving mode discovery and density estimation on synthetic and RNA tasks.
Problem
Long trajectories make credit assignment and exploration hard for amortized samplers (GFlowNets) and entropy-seeking RL, hindering discovery of multiple high-reward modes.
Approach
Online action abstraction: periodically run a BPE-style tokenizer on high-reward sampled action sequences (from sampler plus replay buffer) and add the resulting chunks as new actions, with a policy that embeds variable-size action sets via an LSTM action encoder; two variants, ActionPiece-Increment (add one chunk) and ActionPiece-Replace (add top-M chunks at once).
Key results
- ActionPiece outperforms the Atomic counterpart for the GFlowNet sampler in mode discovery speed across all tasks, while results for A2C and SAC are mixed.
- On the graph environment (7 nodes), ActionPiece-Increment reduces ELBO gap from 0.72 (Atomic) to 0.25 and L1 distance from 1.44 to 1.39.
- GFlowNet-based samplers with ActionPiece-Increment reach chunk occurrence of 1.11 and chunk coverage of 0.69 on L14_RNA1, versus 0.36 and 0.23 for A2C under the same mechanism.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add chunks incrementally instead of using only atomic actions (ablation) | GFlowNet · Graph (7 nodes) | ELBO Gap: 0.72\std{0.39} -> 0.25\std{0.14} | Helped |
| Add chunks incrementally instead of using only atomic actions (ablation) | GFlowNet · Graph (7 nodes) | JSD: 0.38 -> 0.36 | Helped |
| Add chunks incrementally instead of using only atomic actions (ablation) | GFlowNet · Graph (7 nodes) | L1 Distance: $1.44$ -> $\mathbf{1.39}$ | Helped |
| Replace the non-atomic action library with frequent chunks rather than adding chunks incrementally (ablation) | GFlowNet · Graph (7 nodes) | ELBO Gap: 0.72\std{0.39} -> 0.35\std{0.11} | Helped |
| Replace the non-atomic action library with frequent chunks rather than adding chunks incrementally (ablation) | GFlowNet · Graph (7 nodes) | JSD: 0.38 -> 0.40 | Hurt |
| Replace the non-atomic action library with frequent chunks rather than adding chunks incrementally (ablation) | GFlowNet · Graph (7 nodes) | L1 Distance: $1.44$ -> $1.47$ | Hurt |
| Use ActionPiece-Increment instead of atomic actions (ablation) | GFlowNet · FractalGrid, RNA binding, and BitSequence | cumulative modes discovered | Helped |
| Use ActionPiece in place of atomic actions (ablation) | A2C and SAC · FractalGrid | cumulative modes discovered | Helped |
| Use ActionPiece in place of atomic actions (ablation) | A2C and SAC · RNA binding | cumulative modes discovered | Hurt |
| Use ActionPiece-Replace rather than atomic actions (ablation) | A2C · BitSequence | cumulative modes discovered | Helped |
| Use ActionPiece-Increment rather than atomic actions (ablation) | A2C · BitSequence | cumulative modes discovered | Hurt |
| Use ActionPiece-Increment rather than atomic actions (ablation) | GFlowNet · L14_RNA1 | Spearman correlation | No clear effect |
Surprising findings
- The random sampler with ActionPiece achieves strong performance only on simple tasks and degrades on harder problems.
- ActionPiece-Replace, which adds many chunks at once, performs poorly for transfer and density estimation despite being competitive in some settings.
Limitations
- Chunking is performed with BPE for a fixed target distribution; task-conditioned libraries are not learned.
- RL-based samplers (A2C, SAC) show mixed results and can learn overly greedy chunks that hurt exploration.
Experimental setup
Datasets: synthetic FractalGrid RNA binding (L14_RNA1 and related tasks) bit sequence graph cycle environment
Benchmarks: FractalGrid RNA binding bit sequence graph environment
Metrics: cumulative number of modes discovered JSD L1 distance ELBO gap chunk occurrence chunk coverage shortest parse length
Baselines: Atomic sampler MaxEnt-GFlowNet A2C SAC Option-Critic Random sampler
Builds on: trajectory balance byte pair encoding soft actor-critic advantage actor-critic option-critic
Closest papers by meaning
- Reinforcement Learning with Action Chunking (NeurIPS 2025 · similarity 0.88)
- Random Policy Evaluation Uncovers Policies of Generative Flow Networks (ICML 2025 · similarity 0.88)
- Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration (NeurIPS 2026 · similarity 0.87)
- QGFN: Controllable Greediness with Action Values (NeurIPS 2024 · similarity 0.87)
- Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction (ICLR 2025 · similarity 0.87)
- Pre-Training and Fine-Tuning Generative Flow Networks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.