Research map / Vision-language and multimodal / Human and embodied modeling
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FUTURIST, a VAE-free multimodal masked visual sequence transformer that jointly forecasts future semantic segmentation and depth on Cityscapes, reaching new state-of-the-art in future segmentation.
Problem
Predicting future semantic maps (segmentation and depth) for autonomous systems efficiently, without costly VAE tokenizers and without ground-truth labels for every modality.
Approach
Treats next-frame semantic prediction as masked visual modeling in a transformer with hierarchical pixel-to-patch tokenization trained end-to-end (no pretrained VAE), token-wise concatenation fusion of modalities, and a partially-shared plus exclusive multimodal masking strategy.
Key results
- Multimodal FUTURIST achieves 73.9 mIoU (ALL) and 74.9 MO-mIoU short-term on Cityscapes, versus 71.1 for prior best PFA.
- Mid-term forecasting reaches 62.7 mIoU (ALL) and 61.2 MO-mIoU, versus 60.3 and 56.7 for PFA.
- Partially-shared + exclusive masking gives the best results among masking strategies, while fully-masked training is worst (71.2 short-term mIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth as a jointly trained modality instead of using segmentation alone (ablation) | FUTURIST · Cityscapes val, future segmentation | mIoU: 72.5 -> 72.9 (short-term ALL); 61.1 -> 61.6 (mid-term ALL) | Helped |
| Add segmentation as a jointly trained modality instead of using depth alone (ablation) | FUTURIST · Cityscapes val, future depth | δ1: 95.4 -> 95.8 (short-term); 90.6 -> 91.5 (mid-term) | Helped |
| Use independently masked modalities with a shared masking ratio (ablation) | FUTURIST multimodal training · Cityscapes val, short-term segmentation ALL | mIoU: 72.9 -> 72.5 | Hurt |
| Use independently masked modalities with different masking ratios (ablation) | FUTURIST multimodal training · Cityscapes val, short-term segmentation ALL | mIoU: 72.9 -> 71.9 | Hurt |
| Use one shared mask across modalities (ablation) | FUTURIST multimodal training · Cityscapes val, short-term segmentation ALL | mIoU: 72.9 -> 72.5 | Hurt |
| Mask all future tokens across all modalities (ablation) | FUTURIST multimodal training · Cityscapes val, short-term segmentation ALL | mIoU: 72.9 -> 71.2 | Hurt |
| Replace CONCAT fusion with ADD fusion (ablation) | FUTURIST multimodal fusion · Cityscapes val, short-term segmentation ALL | mIoU: 72.5 -> 72.2 | Hurt |
| Increase sequence length from 5 to 7 frames (ablation) | FUTURIST · Cityscapes val, mid-term segmentation ALL | mIoU: 61.6 -> 62.2 | Helped |
| Increase sequence length from 5 to 7 frames (ablation) | FUTURIST · Cityscapes val, mid-term depth | AbsRel ×100: 9.490 -> 9.309 | Helped |
| Train for more epochs (ablation) | FUTURIST · Cityscapes val, future segmentation and depth | prediction performance | Helped |
| Use FUTURIST instead of PFA | Cityscapes future semantic segmentation · Cityscapes val, short-term and mid-term segmentation | mIoU: 71.1 -> 73.9 (short-term ALL); 60.3 -> 62.7 (mid-term ALL) | Helped |
| Predict future semantics directly rather than generate future RGB with VISTA and segment it | Cityscapes future semantic segmentation · Cityscapes val, short-term segmentation ALL | mIoU: 46.3 -> 73.9 | Helped |
Surprising findings
- The VISTA video-diffusion baseline performs worse than the Copy-Last baseline on future segmentation (46.3 vs 55.5 short-term mIoU), because generated frames lose scene semantic structure.
Limitations
- Evaluated only on Cityscapes (and SYNTHIA-Seq in supplement) with pseudo-labels, not true ground truth for depth or context labels.
- Depth evaluation uses DepthAnythingV2 pseudo-annotations rather than real depth ground truth.
Experimental setup
Datasets: Cityscapes SYNTHIA-Seq
Benchmarks: Cityscapes SYNTHIA-Seq
Metrics: mIoU (ALL) MO-mIoU AbsRel delta1
Baselines: PFA F2MF LSTM M2M DeformF2F Dil10-S2S VISTA Copy-Last Oracle
Builds on: MaskGIT VideoGPT Segmenter DepthAnythingV2 Pre-LN transformer
Closest papers by meaning
- DINO-Foresight: Looking into the Future with DINO (NeurIPS 2025 · similarity 0.87)
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (NeurIPS 2025 · similarity 0.86)
- FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs (ICML 2026 · similarity 0.86)
- Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting (ICML 2025 · similarity 0.86)
- FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities (NeurIPS 2025 · similarity 0.85)
- VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.