Research map / Vision-language and multimodal / Robot action and navigation

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

ICLR 2025 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

LLaRA converts behavior-cloning trajectories into conversation-style instruction data with image-coordinate actions and adds six self-supervised auxiliary datasets to finetune LLaVA-1.5-7B into a robot policy.

Problem

Adapting pretrained vision-language models into robot action policies is hard when robot demonstration data is limited, and conversation-style instruction data for robotics is under-explored.

Approach

Formulate robot actions as text with 2D image coordinates (normalized to [0,1]) inside visual instruction-tuning conversations, and enhance the data with six auxiliary self-supervised instruction datasets (localization, detection, action/future prediction, spatial and temporal relations) derived from existing trajectories without extra action labels.

Key results

• In the real world with finetuning on xArm-Det, D-inBC + Aux (C) + OD reaches 91.6% average success... more precisely, the xArm-Action finetuned variant reaches 91.6% average success versus 31.6% for GPT-4o in zero-shot.
• Compared to VIMA's most capable released model (trained on 660k trajectories), LLaRA's best model is reported to perform better while using about 12% of the expert trajectories and only the front view.
• Auxiliary datasets give large gains when training data is limited, e.g. real-world finetuning on xArm-Det rises from 0% (inBC) to 88.3% with Aux (D).

Measured effects reported in the paper

Surprising findings

• RT-2 Style baselines scored 0% in all real-world settings, including with VIMA-660k pretraining, showing that token-based action outputs struggle with limited data.
• Auxiliary data gains diminish or even hurt when the BC dataset is large (VIMA-80k) and in zero-shot real-world evaluation, where aux data caused overfitting to the VIMA domain.

Limitations

• Real-world evaluation uses only 20 episodes per setting and a single robot arm with a fixed camera and linear calibration mapping.
• Auxiliary data can hurt performance in zero-shot settings and when the BC dataset is large, requiring careful tuning of its amount.
• Only single-image inputs are used; multiple reference images are converted to text via object detection.

Experimental setup

Datasets: VIMA-Bench (VIMA dataset subsets) xArm-Det xArm-Action Instruct-BC (inBC) Description-Instruct-BC (D-inBC) auxiliary datasets (localization, detection, action prediction, future prediction, spatial relationship, temporal relati

Benchmarks: VIMA-Bench (L1-L3) real-world xArm tasks T1-T3 real-world generalization tasks with distractors and unseen T4

Metrics: success rate

Baselines: RT-2 Style VIMA GPT-4o OpenVLA

Builds on: LLaVA RT-2 VIMA-Bench OWLv2

Closest papers by meaning

• FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.91)
• Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
• NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
• ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
• VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation (ICLR 2025 · similarity 0.90)
• PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)