Research map / Reinforcement learning / Visual and representation RL
Unlock the Intermittent Control Ability of Model Free Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARS, a conditional-VAE latent representation of multi-step action sequences (with an action-transition-scale condition and a dynamics residual prediction loss) that lets model-free RL such as TD3 plan advance action sequences under intermittent interaction.
Problem
Reinforcement learning under intermittent control, where communication between decision maker and executor is blocked for several steps, so the agent must output multi-step action sequences without seeing intermediate states; naive multi-step outputs suffer from action-space explosion and unnatural
Approach
Learn a compact, decodable latent space of multi-step action sequences with a conditional VAE conditioned on an action transition scale, plus an unsupervised state-residual prediction loss for dynamic semantic smoothness; the RL policy then selects latent actions and scales that are decoded into action sequences.
Key results
- On random-interval simulated tasks, MARS-TD3 outperforms Frameskip-TD3 and Multistep-TD3 across Hopper, Ant, Walker, HalfCheetah and 2D mazes, and is comparable to Perfect-TD3 in most tasks.
- On the real-world 6-DoF arm grasping task, MARS + TD3-BC reaches 74.2% grasp success and 70.8% motion smoothness versus 61.5% GSR for Multistep + TD3-BC and 34.1% for Vanilla TD3-BC.
- In Table 5, MARS-TD3 reaches 3762.7 on fixed-interval Hopper versus 1359.7 for TD3-multistep and 788.1 for Dreamer-v2, indicating MBRL baselines underperform.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MARS instead of multistep action output | TD3-BC · Robot arm grasping | GSR (%): 61.5 -> 74.2 | Helped |
| Use MARS instead of vanilla TD3-BC | TD3-BC · Robot arm grasping | MSR (%): 22.5 -> 70.8 | Helped |
| Use MARS instead of TD3-multistep | TD3 · Hopper (fixed-Intermittent) | return: 1359.7 ± 258.3 -> 3762.7 ± 371.4 | Helped |
| Use MARS instead of TD3-multistep | TD3 · Ant (random-Intermittent) | return: 2795.4 ± 264.1 -> 3105.7 ± 412.6 | Helped |
| Add action transition scale conditioning to the representation (ablation) | MARS latent action representation · Hopper, random Intermittent-MDP | return | Helped |
| Add state dynamic prediction to the representation (ablation) | MARS latent action representation · Hopper, random Intermittent-MDP | return | Helped |
| Combine action transition scale conditioning and state dynamic prediction (ablation) | MARS latent action representation · Hopper, random Intermittent-MDP | return | Helped |
| Reduce warm-up steps from 400000 to 100000 (ablation) | MARS-TD3 · Hopper | return: 3219.1 -> 3086.4 | Hurt |
| Reduce warm-up steps from 400000 to 100000 (ablation) | MARS-TD3 · Ant | return: 4305.7 -> 4025.6 | Hurt |
| Reduce warm-up steps from 400000 to 100000 (ablation) | MARS-TD3 · Walker | return: 4961.3 -> 4792.6 | Hurt |
| Reduce warm-up steps from 400000 to 100000 (ablation) | MARS-TD3 · HalfCheetah | return: 6593.2 -> 6071.2 | Hurt |
| Reduce warm-up steps from 100000 to 30000 (ablation) | MARS-TD3 · 2dmaze-medium | return: 127.8 -> 118.5 | Hurt |
Surprising findings
- Model-based baselines (TD-MPC, Dreamer-v2) perform worse than or similar to direct multi-step TD3 under intermittent control, attributed to compounding model error.
- MARS representation learning is reported to be insensitive to data quality; random-policy data suffices for training the representation.
Limitations
- Performance degrades for very long intermittent intervals, since the VAE struggles to represent very long action sequences.
- Representation model is an MLP VAE; the authors note it may need stronger architectures (e.g., transformer VAE) for extremely long sequences.
Experimental setup
Datasets: D4RL 2dmaze medium/hard MuJoCo Hopper, Ant, Walker, HalfCheetah
Benchmarks: MuJoCo Hopper MuJoCo Ant MuJoCo Walker MuJoCo HalfCheetah D4RL 2dmaze medium D4RL 2dmaze hard real-world robot arm grasping
Metrics: average episode reward grasp success rate declutter rate motion smooth ratio
Baselines: Perfect-TD3 Frameskip-TD3 Multistep-TD3 TD3-BC Dense observation + TD3-BC TD-MPC Dreamer-v2
Builds on: TD3 TD3-BC Conditional VAE Frameskip TD-MPC
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Sleeping Reinforcement Learning (ICML 2025 · similarity 0.87)
- Improving planning and MBRL with temporally-extended actions (NeurIPS 2025 · similarity 0.86)
- Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks (ICLR 2024 · similarity 0.85)
- Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL (NeurIPS 2024 · similarity 0.85)
- Select before Act: Spatially Decoupled Action Repetition for Continuous Control (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.