Research map / Architectures and efficiency / Attention and transformers
Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoIn, a lightweight convolutional injector module that adds locality and translation-equivariance biases to pretrained ViTs (CLIP, MVP, VC-1) via cross-attention, improving behavior-cloning finetuning across 12 visuo-motor control tasks.
Problem
Pretrained ViTs have weak inductive biases (spatial locality, translation equivariance) that may hinder their adaptation for visuo-motor control, where such biases are important.
Approach
A lightweight CNN encoder produces a multi-scale feature pyramid that is injected into the ViT's patch tokens through a deformable cross-attention layer (injector only, no extractor), adapting pretrained ViTs for control without retraining from scratch.
Key results
- CoIn with CLIP ViT-B improves mean success from 59.6 to 70.9 (+11.3) across 12 tasks under full finetuning.
- CoIn with MVP and VC-1 improves mean success by +1.7 (84.5 to 86.2) and +3.9 (80.7 to 84.6) respectively.
- CoIn adds about 3.1M trainable parameters (3.6% of ViT-B/16) and outperforms AdaptFormer and RoboAdapter adapter baselines under full finetuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CoIn to finetuned CLIP | CLIP ViT-B/16 · 12 control tasks | Mean Success: 59.6 -> 70.9 (+11.3) | Helped |
| Add CoIn to finetuned MVP | MVP ViT-B/16 · 12 control tasks | Mean Success: 84.5 -> 86.2 (+1.7) | Helped |
| Add CoIn to finetuned VC-1 | VC-1 ViT-B/16 · 12 control tasks | Mean Success: 80.7 -> 84.6 (+3.9) | Helped |
| Replace MLP adapter with CoIn | CLIP full finetuning · 12 control tasks | Mean: 59.6 -> 70.9 | Helped |
| Use multi-scale feature pyramid instead of only F2 (ablation) | CLIP + CoIn · 12 control tasks | Mean: 70.4 -> 70.9 | Helped |
| Use multi-scale feature pyramid instead of only F2 (ablation) | VC-1 + CoIn · 12 control tasks | Mean: 83.7 -> 84.6 | Helped |
| Add weight decay during finetuning (ablation) | VC-1 ViT-B · 12 control tasks | Mean Success: 55.1 -> 58.9 | Helped |
| Add cosine learning rate decay (ablation) | VC-1 ViT-B with weight decay · 12 control tasks | Mean Success: 58.9 -> 76.5 | Helped |
| Add layer-wise learning rate decay (ablation) | VC-1 ViT-B with weight decay and cosine LR · 12 control tasks | Mean Success: 76.5 -> 80.7 | Helped |
| Finetune VC-1 without weight decay, cosine LR, or LLDR rather than keeping it frozen (ablation) | VC-1 ViT-B · 12 control tasks | Mean Success: 65.9 -> 55.1 | Hurt |
| Add CoIn to VC-1 ViT-B (ablation) | VC-1 ViT-B/16 · 12 control tasks | Mean: 80.7 -> 84.6 | Helped |
| Add CoIn to VC-1 ViT-L (ablation) | VC-1 ViT-L/14 · 12 control tasks | Mean: 83.4 -> 84.5 | Helped |
Surprising findings
- Naively finetuning VC-1 end-to-end deteriorates performance versus the frozen encoder (65.9 to 55.1), reversed by weight decay, cosine LR schedule, and layer-wise LR decay.
- ViT-B with CoIn (84.6) outperforms the larger ViT-L without CoIn (83.4) despite far fewer parameters.
Limitations
- Evaluated only in simulation; no real-world robot experiments.
- Only behavior cloning is studied; extension to reinforcement learning is left to future work.
- Gains for MVP and VC-1 are modest compared with CLIP.
Experimental setup
Datasets: Adroit expert demonstrations MetaWorld expert demonstrations DMC expert demonstrations
Benchmarks: Adroit MetaWorld DMC
Metrics: success rate normalized episode return mean success
Baselines: CLIP finetuned MVP finetuned VC-1 finetuned VIP R3M AdaptFormer RoboAdapter ViT-Adapter
Builds on: ViT-Adapter deformable attention AdaptFormer MAE
Closest papers by meaning
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.85)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.85)
- Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning (CVPR 2025 · similarity 0.85)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.