Research map / Reinforcement learning / Visual and representation RL

Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Macro Action Quantization (MAQ), which distills human demonstrations into a VQ-VAE codebook of macro actions and lets standard RL agents (IQL, SAC, RLPD) choose among them, yielding more human-like Adroit behavior.

Problem

RL agents often behave unnaturally compared with humans (jerky, spinning, shaking), while imitation learning limits task performance; human-likeness needs to be achieved without sacrificing success.

Approach

Formulate human-likeness as trajectory optimization solved with receding-horizon control over action segments; quantize human action segments (macro actions) with a conditional VQ-VAE so the RL policy selects discrete, human-like macro actions conditioned on state, integrable with any RL algorithm.

Key results

• MAQ+RLPD reaches 71% human-likeness win rate in the ranking test, comparable to human demonstrations at 74%, versus 38% for RLPD.
• On Door, DTW_s rises from 0.43 (IQL) to 0.84 (MAQ+IQL) and from -0.39 (SAC) to 0.80 (MAQ+SAC), with MAQ+IQL success 0.93 vs 0.16 for IQL.
• In the Turing test, MAQ+RLPD fools evaluators 39% of the time on average versus 24% for RLPD.

Measured effects reported in the paper

Surprising findings

• Hammer success for MAQ+RLPD (0.56) is lower than RLPD (1.00) at the 200-step horizon; raising the horizon to 450 steps raised MAQ+RLPD success to 0.72, suggesting human-like slow behavior is penalized by the episode limit.

Limitations

• Depends on availability and quality of human demonstrations (only 25 trajectories per task).
• Evaluated only on four Adroit tasks with a primitive action space of Shadow Hand joint positions; generalization to more complex domains is untested.
• Human-likeness similarity metrics do not capture full semantics of human-like intent, as acknowledged in the appendix.

Experimental setup

Datasets: D4RL Adroit human demonstrations (Door, Hammer, Pen, Relocate)

Benchmarks: D4RL Adroit

Metrics: DTW state DTW action Wasserstein distance state Wasserstein distance action success rate Turing test win rate human-likeness ranking win rate

Baselines: BC IQL SAC RLPD

Builds on: VQ-VAE Semi-Markov Decision Process IQL RLPD Discrete SAC

Closest papers by meaning

• Reinforcement Learning from Imperfect Corrective Actions and Proxy Rewards (ICLR 2025 · similarity 0.88)
• Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning (ICLR 2025 · similarity 0.87)
• DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.87)
• GUIDE: Real-Time Human-Shaped Agents (NeurIPS 2024 · similarity 0.87)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.87)
• Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.87)