Research map / Vision-language and multimodal / Robot action and navigation
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OTTER, a VLA that keeps CLIP frozen and extracts text-aware visual tokens (via ClearCLIP-style attention features and text-patch softmax) before a small causal transformer policy, improving zero-shot generalization on LIBERO and real Franka tasks.
Problem
Fine-tuning pre-trained VLMs for robot control degrades their vision-language alignment, hurting generalization to unseen objects and tasks.
Approach
Freeze CLIP and use the text-visual similarity softmax over patch features (from the last attention block output, per ClearCLIP) to select and pool task-relevant visual tokens per text token, feeding these into a small causal transformer alongside text and proprioception tokens.
Key results
- On real-world unseen pick-and-place tasks, OTTER reaches 62% success vs 9% for finetuned OpenVLA and 4% for DFP-OTTER; OTTER-OXE reaches 73%, above pi0-Fast-Droid at 61%.
- Across four real primitives on unseen tasks, OTTER-OXE-L achieves 77% mean success while finetuned Octo and OpenVLA reach 4% and 0.6%.
- On LIBERO unseen tasks, OTTER reaches 61% vs 48% for finetuned OpenVLA and 26% for finetuned Octo, while matching in-distribution performance (84%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text-aware visual feature extraction with frozen CLIP rather than direct feature passing | OTTER · Real pick-and-place unseen tasks | success rate: 4% $\pm$ 1.6% -> 62% $\pm$ 4.2% | Helped |
| Pre-train on OXE before fine-tuning | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 73% $\pm$ 2.8% | Helped |
| Increase policy network capacity | OTTER · Real multi-primitive unseen tasks | mean success rate: 67% $\pm$ 3.8% -> 71% $\pm$ 3.5% | Helped |
| Remove embodiment feature f_e (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 29% $\pm$ 4.3% | Hurt |
| Remove text token f'_l (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 53% $\pm$ 4.6% | Hurt |
| Replace frozen CLIP vision encoder with a vision encoder trained from scratch (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 11% $\pm$ 2.5% | Hurt |
| Fine-tune CLIP end to end instead of keeping it frozen (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 15% $\pm$ 3.9% | Hurt |
| Use the CLIP CLS token instead of text-aware visual feature extraction (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 6% $\pm$ 0.8% | Hurt |
| Replace text-aware visual feature extraction with standard cross-attention pooling (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 2% $\pm$ 0.5% | Hurt |
| Pre-train on human videos (ablation) | OTTER · Real pick-and-place unseen tasks | success rate: 62% $\pm$ 4.2% -> 70% $\pm$ 4.5% | Helped |
| Scale CLIP from ViT-B/32 to ViT-L/14 | OTTER · Real pick-and-place tasks | success rate: +27.5% training; +39.3% unseen | Helped |
| Remove CLIP vision encoder and train vision from scratch (ablation) | OTTER · LIBERO unseen tasks | success rate: 61% $\pm$ 1.1% -> 29% $\pm$ 0.9% | Hurt |
Surprising findings
- Fine-tuning CLIP inside the VLA drops unseen real-world success from 62% to 15% and degrades ImageNette top-1 accuracy from 92.2% to 10.2%.
- Direct feature passing with frozen CLIP (DFP-OTTER) fails to generalize to unseen tasks (4%), showing the extraction step rather than frozen encoding alone drives gains.
Limitations
- Scaling to morphologies not parameterized by SE(3), such as multi-finger hands, is not addressed.
- Long-horizon tasks and more complex scenes are not extensively explored.
- Human-video pre-training gives only modest benefit and the human data differs greatly in viewpoint and proprioception.
Experimental setup
Datasets: LIBERO Open X-Embodiment DexYCB HOI4D DS-PnP DS-ALL
Benchmarks: LIBERO-Spatial LIBERO-Object LIBERO-Goal LIBERO unseen tasks real-world Franka pick-and-place, pouring, drawer, poking
Metrics: success rate
Baselines: Octo OpenVLA pi0-Fast-Droid DFP-OTTER OTTER (Finetune CLIP)
Builds on: CLIP ClearCLIP Octo OpenVLA LIBERO
Closest papers by meaning
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.