Research map / Vision-language and multimodal / Robot action and navigation

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FineVLA, an open framework that builds 10-dimension fine-grained process-level instructions for 47K trajectories, a robotic video benchmark, a fine-tuned VLM annotator, and studies FG:Raw instruction mixing for steerable VLA policies.

Problem

VLA policies are trained with coarse goal-level instructions that leave execution details (actor, approach, contact, motion direction) unspecified, limiting steerability and robotic video understanding.

Approach

Unify 10 open robot datasets, cluster redundant demonstrations with DTW to select representatives, annotate them with ten-dimensional fine-grained process descriptions (VLM draft plus human verification), and train policies with controlled mixtures of fine-grained and raw instructions while holding actions and observations fixed.

Key results

• On RoboTwin, AlohaMix-OFT with FG:Raw 1:1 reaches 86.8%/82.5% Easy/Hard vs 71.8%/71.4% for Raw-only.
• In real-world dual-arm tests, FG:Raw 1:1 scores 62.7/100 vs 49.9 for Raw-only; largest factor gains on pose (+23), color (+18), approach (+18).
• RoboFine-VLM reaches 68.2% VQA accuracy, beating GPT-5.4 (60.2%), and 82.2% hard-setting caption overall.

Surprising findings

• FG-only policies underperform mixed FG:Raw ratios, so pure fine-grained language is not optimal despite beating Raw-only.
• Out-of-distribution actor-target binding remains largely unsolved, reaching only 10/100 at best.

Limitations

• RoboFine-VLM does not fully remove human verification cost.
• Real-world validation limited to one tabletop dual-arm platform and a small set of steerability tasks.
• Compositional actor-target generalization remains unsolved; anti-hallucination measured only over action steps.

Experimental setup

Datasets: BridgeData-V2 BC-Z RT-1 Galaxea RoboMIND-V1 RoboMIND-V2 RoboCOIN RH20T RDT DROID RoboTwin

Benchmarks: RoboFine-Bench RoboTwin Real-world Steerability Suite

Metrics: VQA accuracy caption Consistency caption Coverage caption Anti-Hallucination success rate partial-completion score (0-100)

Baselines: Raw-only FG-only Gemini-3.1-Pro GPT-5.4 Doubao-Seed-2.0-Pro Qwen3.5-Plus Qwen3-VL-Plus

Builds on: OpenVLA-OFT GR00T N1.5 RoboTwin StarVLA

Closest papers by meaning

• ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
• Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
• NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.92)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (ICLR 2025 · similarity 0.91)
• PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
• RotVLA: Rotational Latent Action for Vision-Language-Action Model (NeurIPS 2026 · similarity 0.91)