Research map / Vision-language and multimodal / Robot action and navigation
FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSDrive, a VLA that generates a unified future frame (lane dividers, 3D boxes, background) as visual spatio-temporal chain-of-thought, then plans trajectories conditioned on it, trained with a unified understanding/generation pre-training on an expanded visual vocabulary.
Problem
Textual chain-of-thought in VLA-based end-to-end driving compresses visual information into symbols, blurring spatio-temporal relations and creating a modality gap between perception and planning.
Approach
Use the VLA itself as a world model that autoregressively generates a single image-form future frame with explicit lane dividers and 3D boxes drawn on it, then act as an inverse-dynamics planner conditioned on current observations and this visual CoT; activate visual generation in an existing MLLM by extending its vocabulary with MoVQGAN tokens, using a progressive easy-to-hard generation curricul
Key results
- On nuScenes planning with no ego status, FSDrive (Qwen2-VL-2B) reaches 0.96 m average L2 under ST-P3 metrics and 0.40% collision under UniAD metrics, beating Doe-1 (0.70 m, 0.21%) under the same ST-P3 metric.
- FSDrive achieves FID 10.1 on nuScenes future-frame generation at 128x192, better than GEM (10.5) and Drive-WM (15.8) despite lower resolution.
- On NAVSIM navtest, FSDrive scores PDMS 85.1, above LAW (84.6) among camera-only methods; on DriveLM GVQA it reaches a final score of 0.57 vs OmniDrive 0.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FSDrive instead of LAW | camera-only planning on NAVSIM navtest · NAVSIM navtest | PDMS: 84.6 -> 85.1 | Helped |
| use FSDrive instead of GEM | future-frame generation on nuScenes · nuScenes | FID: 10.5 -> 10.1 | Helped |
| add VQA pre-training (ablation) | Qwen2-VL-2B driving model · nuScenes | L2 Avg. (m), UniAD metrics: 1.22 -> 1.19 | Helped |
| add future-frame pre-training (ablation) | Qwen2-VL-2B driving model · nuScenes | L2 Avg. (m), UniAD metrics: 1.22 -> 1.02 | Helped |
| add future 3D-detection pre-training (ablation) | Qwen2-VL-2B driving model · nuScenes | L2 Avg. (m), UniAD metrics: 1.22 -> 1.17 | Helped |
| add future lane-divider pre-training (ablation) | Qwen2-VL-2B driving model · nuScenes | Collision Avg. (%), UniAD metrics: 0.67 -> 0.65 | Helped |
| combine VQA, future frames, 3D detection, and lane-divider pre-training (ablation) | Qwen2-VL-2B driving model · nuScenes | L2 Avg. (m), UniAD metrics: 1.22 -> 0.98 | Helped |
| replace no CoT with text CoT (ablation) | Qwen2-VL-2B trajectory planning · nuScenes | Collision Avg. (%), UniAD metrics: 0.58 -> 0.53 | Helped |
| replace text CoT with image-text CoT (ablation) | Qwen2-VL-2B trajectory planning · nuScenes | Collision Avg. (%), UniAD metrics: 0.53 -> 0.50 | Helped |
| replace no CoT with spatio-temporal CoT (ablation) | Qwen2-VL-2B trajectory planning · nuScenes | Collision Avg. (%), UniAD metrics: 0.58 -> 0.40 | Helped |
| increase future-frame pre-training data (ablation) | Qwen2-VL-2B visual generation · nuScenes | FID: 29.4 -> 16.2 -> 12.7 | Helped |
| add progressive generation to ~200k pre-training data (ablation) | Qwen2-VL-2B visual generation · nuScenes | FID: 12.7 -> 10.1 | Helped |
Surprising findings
- Pure text CoT and image-text CoT give only marginal L2 gains; the main benefit is in collision rate, with spatio-temporal CoT reducing average collision by about 31%.
- Pure VQA pre-training had negligible effect on planning metrics, while future-frame generation pre-training gave the largest L2 improvement.
Limitations
- Only front-view frames are generated, not surround views.
- Generated frame quality is limited by small pre-training data (~200k samples) and low resolution (128x192).
- Planning gains with ego status are reported but the paper prioritizes non-ego-status settings; results depend on the chosen metric computation (ST-P3 vs UniAD).
Experimental setup
Datasets: nuScenes OmniDrive-nuScenes DriveLM GVQA NAVSIM
Benchmarks: nuScenes NAVSIM DriveLM GVQA
Metrics: L2 (m) collision rate (%) FID PDMS BLEU_1 ROUGE_L CIDEr ChatGPT score accuracy match
Baselines: ST-P3 VAD UniAD BEV-Planner PreWorld ELM FeD OccWorld Doe-1 RDA-Driver EMMA OmniDrive
Builds on: Qwen2-VL Lumina-mGPT VLIPP OmniDrive MoVQGAN
Closest papers by meaning
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.91)
- OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.91)
- Explicit Geometric Chain-of-Thought for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.91)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.