Research map / Reinforcement learning / Visual and representation RL
Vintix: Action Model via In-Context Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Vintix trains a 300M-parameter cross-domain Algorithm Distillation transformer on 87 noise-distilled RL tasks (MuJoCo, Meta-World, Bi-DexHands, Industrial-Benchmark) to perform in-context RL with cold-start self-correction at inference.
Problem
Scaling in-context reinforcement learning beyond toy, single-domain tasks to a cross-domain generalist action model that improves at inference time from reward feedback.
Approach
Extends epsilon-greedy noise distillation to continuous action spaces (linear mixture with uniform noise, annealed over the trajectory) to cheaply collect learning histories, then trains a single task-agnostic transformer via Algorithm Distillation on a shared cross-domain dataset, stacking action, reward and observation into one token.
Key results
- Vintix reaches near-demonstrator performance on training tasks via cold-start many-shot inference, e.g. 0.97 average expert-normalized on MuJoCo and 0.95 on Meta-World (AD) versus 0.8 for expert distillation.
- Reported to outperform JAT and REGENT on Meta-World and MuJoCo by significant margins under demonstrator normalization (with caveat that normalization was re-based).
- On held-out tasks it is weak: 47% of expert score on Meta-World Door-Unlock, 31% on Bi-DexHands Door-Open-Inward, and random-level on Bin-Picking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Algorithm Distillation on noise-distilled trajectories instead of expert-only distillation (ablation) | Vintix · MuJoCo training tasks | average expert-normalized score: 0.8 -> 0.97 | Helped |
| Use Algorithm Distillation on noise-distilled trajectories instead of expert-only distillation (ablation) | Vintix · Meta-World training tasks | average expert-normalized score: 0.8 -> 0.95 | Helped |
| Add reward signals during training (ablation) | Vintix Algorithm Distillation · Meta-World training tasks | asymptotic performance | Helped |
| Add reward signals during training (ablation) | Vintix Algorithm Distillation · Meta-World training tasks | convergence speed | Helped |
| Use expert-only demonstrations rather than trajectories containing policy-improvement structure (ablation) | Vintix · MuJoCo and Meta-World | asymptotic performance: 0.97 and 0.95 vs. 0.8 | Hurt |
| Increase the number of inference episodes (shots) from a cold start | Vintix · Training tasks across four domains | return | Helped |
| Apply inference-time self-correction | Vintix · MuJoCo and Meta-World training tasks | demonstrator-normalized return | Helped |
| Evaluate on unseen viscosity, gravity, and Industrial-Benchmark setpoint variations | Vintix · MuJoCo and Industrial-Benchmark parametric variations | performance relative to demonstrators | Mixed |
| Apply cold-start in-context adaptation to unseen tasks | Vintix · Meta-World Door-Unlock | mean expert-normalized score: 47% | Hurt |
| Apply cold-start in-context adaptation to unseen tasks | Vintix · Bi-DexHands Door-Open-Inward | demonstrator performance: 31% | Hurt |
| Apply cold-start in-context adaptation to unseen tasks | Vintix · Meta-World Bin-Picking and Bi-DexHands Hand-Kettle | performance relative to random baseline | Hurt |
Surprising findings
- Removing reward signals from training degrades inference-time improvement, indicating reward feedback is needed for in-context RL behavior.
- Expert distillation on the same architecture converges to lower asymptotic performance (0.8) than algorithm distillation on noise-distilled histories.
Limitations
- Fails to generalize to significantly new tasks; performance on held-out tasks is often near random.
- Restricted to continuous vector observations and actions with dimensionality-group encoders; no task identifier, and tasks with unseen dimensionality (Hand-Over) cannot be handled.
- Normalization relies on re-scored demonstrators, so comparison to JAT and REGENT is not fully like-for-like; does not exceed demonstrator performance.
Experimental setup
Datasets: MuJoCo Meta-World Bi-DexHands Industrial-Benchmark
Benchmarks: Meta-World ML45 training tasks Bi-DexHands ML20 training tasks MuJoCo training tasks Industrial-Benchmark setpoints 0-75 MuJoCo parametric variations Industrial-Benchmark setpoints 80-100 Meta-World and Bi-DexHands held-out tasks
Metrics: demonstrator-normalized return IQM
Baselines: JAT REGENT Expert Distillation (ED) AD without rewards
Builds on: Algorithm Distillation AD-epsilon noise distillation Decision Transformer JAT ALiBi
Closest papers by meaning
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds (NeurIPS 2025 · similarity 0.90)
- Towards Provable Emergence of In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Zero-shot Model-based Reinforcement Learning using Large Language Models (ICLR 2025 · similarity 0.89)
- Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning (ICLR 2025 · similarity 0.89)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Behavior-agnostic Task Inference for Robust Offline In-context Reinforcement Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.