Research map / Vision-language and multimodal / Human and embodied modeling
Task-Conditioned Adaptation of Visual Features in Multi-Task Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen MAE-pretrained VC-1 ViT with task-conditioned visual adapters and a single behavior-cloned multi-task policy, and few-shot generalizes to unseen tasks by optimizing a task embedding from demonstrations.
Problem
Single multi-task robot policies built on generally pre-trained vision models use fixed visual features that are not adapted to the task; the paper asks whether task-conditioned adaptation of perception helps multi-task and few-shot control.
Approach
Insert middle and top adapters, conditioned on a learned task embedding, into a frozen pre-trained ViT (VC-1) so that visual features are modulated per task without fine-tuning backbone weights; for unseen tasks, estimate the task embedding by minimizing the policy's action loss on a few demonstrations.
Key results
- Multi-task policy with middle and top conditioned adapters reaches 54.8% average test success across 12 known tasks (row f), vs 46.3% for single-task VC-1 policies (row a).
- Random or randomly picked task embeddings collapse performance to about 4% average, showing the learned embedding is necessary.
- Few-shot: with task embeddings optimized from 5 demonstrations, the policy adapts to many of 15 unseen MetaWorld tasks without weight finetuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add task-conditioned middle and top visual adapters (ablation) | Multi-task policy with VC-1 · CortexBench known tasks, 12-task average (test) | performance: 42.5 ± 1.2 -> 54.8 ± 1.2 | Helped |
| Add task-conditioned middle adapters (ablation) | Multi-task policy with VC-1 · CortexBench known tasks, 12-task average (test) | performance: 44.2 ± 1.8 -> 48.8 ± 1.3 | Helped |
| Add a conditioned top adapter to conditioned middle adapters (ablation) | Multi-task policy with VC-1 · CortexBench known tasks, 12-task average (test) | performance: 48.8 ± 1.3 -> 54.8 ± 1.2 | Helped |
| Condition the top adapter instead of using a non-conditioned top adapter (ablation) | Multi-task policy with conditioned middle adapters and VC-1 · CortexBench known tasks, 12-task average (test) | performance: 52.5 ± 1.1 -> 54.8 ± 1.2 | Helped |
| Use the learned task embedding rather than a random vector (ablation) | Conditioned adapters with VC-1 · CortexBench known tasks, 12-task average (test) | performance: 3.8 ± 0.4 -> 54.8 ± 1.2 | Helped |
| Use the correct task embedding rather than a randomly picked learned embedding (ablation) | Conditioned adapters with VC-1 · CortexBench known tasks, 12-task average (test) | performance: 4.6 ± 2.5 -> 54.8 ± 1.2 | Helped |
| Use task-conditioned adapters and a multi-task policy instead of separate single-task policies | VC-1 policies · CortexBench known tasks, 12-task average (test) | performance: 46.3 ± 0.4 -> 54.8 ± 1.2 | Helped |
Surprising findings
- Conditioning the adapters on the task embedding helps, but the multi-task policy without any adapters already outperforms the per-task single-task VC-1 policies on average is not claimed; the multi-task row (b) without adapters is worse than single-task policies, so gains come from adapters.
Limitations
- Few-shot success is low on several unseen tasks (e.g., 0% on button-press-wall), and failures often come from incomplete execution of sparse-success tasks.
- Unseen-task demonstrations are generated by TD-MPC2 policies rather than humans, and tasks are filtered to keep the goal visible in camera FOV.
Experimental setup
Datasets: CortexBench trajectories (Adroit, DeepMind Control Suite, MetaWorld) TD-MPC2-generated MetaWorld demonstrations for unknown tasks
Benchmarks: CortexBench Adroit DeepMind Control Suite MetaWorld
Metrics: rollout success rate normalized return
Baselines: VC-1 single-task policies (Majumdar et al. 2023) Non-adapted multi-task policy Non-conditioned adapters Random task embedding
Builds on: VC-1 Adapters (Houlsby et al.) Visual adapters for ViT (Sharma et al.) MAE TD-MPC2
Closest papers by meaning
- Selective Visual Representations Improve Convergence and Generalization for Embodied AI (ICLR 2024 · similarity 0.87)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning (ICLR 2025 · similarity 0.87)
- TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.86)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.86)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.