Research map / Vision-language and multimodal / Robot action and navigation
TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes visual trace prompting, overlaying CoTracker-derived point trajectories on robot observations, and fine-tunes OpenVLA (7B) and a 4B Phi-3-Vision VLA on ~150K trajectories to improve spatial-temporal awareness in manipulation.
Problem
VLA policies react to current observations and lack awareness of their past spatial movements, limiting performance and generalization on manipulation tasks.
Approach
Track dense points with CoTracker over a short history window, keep the actively moving trajectories, and overlay them on the initial frame as a visual prompt fed alongside the original image, with trace-dropout so the model still works when tracking fails.
Key results
- TraceVLA (7B) reaches 47.7% overall SimplerEnv success vs 40.2% for OpenVLA, a 7.5-point gain.
- On real WidowX tasks, TraceVLA achieves 8/10 on unseen pick-place corn versus 1/10 for OpenVLA.
- On LIBERO multitask suites, TraceVLA averages 74.8% vs 70.6% for finetuned OpenVLA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual trace prompting | OpenVLA · SimplerEnv Google robot tasks | overall performance: 40.2% -> 47.7% | Helped |
| Add visual trace annotations during finetuning | OpenVLA · LIBERO average | success rate: 70.6% ± 0.4% -> 74.8% ± 0.4% | Helped |
| Replace visual trace prompts with six historical image observations (ablation) | OpenVLA finetuned · SimplerEnv | overall performance: -6% | Hurt |
| Use text-based point movement traces instead of visual traces (ablation) | OpenVLA · SimplerEnv | average performance: +2.4%; +6.4% | Mixed |
| Use a visual trace length of N=3 (ablation) | TraceVLA · SimplerEnv | performance: 3.2% | Helped |
| Use a visual trace length of N=12 (ablation) | TraceVLA · SimplerEnv | overall performance | Hurt |
| Vary visual trace thickness, transparency, or color within the tested ranges (ablation) | TraceVLA · SimplerEnv Average Success Rate | success rate: 47.2%, 47.7%, 47.8%; 47.7%, 47.3%; 47.7%, 47.3% | No clear effect |
| Finetune OpenVLA on the same dataset without visual trace prompting (ablation) | OpenVLA · SimplerEnv | average success rate: +1.1% | Helped |
| Finetune OpenVLA-Phi3 on the same dataset without visual trace prompting (ablation) | OpenVLA-Phi3 · SimplerEnv | average success rate: -0.3% | Hurt |
Surprising findings
- Finetuning OpenVLA on 6 stacked history frames reduces SimplerEnv performance by about 6 points, while visual traces help.
- Finetuning OpenVLA-Phi3 on the same subset slightly degrades performance (-0.3%).
Limitations
- Relies on CoTracker, which can lose tracks (e.g., under poor lighting), requiring trace dropout and periodic re-querying.
- Additional inference cost and extra image tokens, plus memory overhead during training.
- Real-robot evaluation uses only 4 tasks with 10 trials each, and simulation results come from a single benchmark family (SimplerEnv Google robot).
Experimental setup
Datasets: BridgeData-v2 Google RT-1 robot dataset Open X-Embodiment LIBERO
Benchmarks: SimplerEnv (Google robot, visual matching and variant aggregation) WidowX-250 real robot tasks LIBERO-Spatial/Object/Goal/Long
Metrics: success rate
Baselines: OpenVLA OpenVLA-Phi3 Octo-Base RT1-X text trace prompting OpenVLA with 6-frame history
Builds on: OpenVLA CoTracker Prismatic VLM Phi-3-Vision
Closest papers by meaning
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.90)
- FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies (NeurIPS 2026 · similarity 0.90)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.90)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.89)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.