Research map / Vision-language and multimodal / Multimodal LLMs

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FutureOmni, a 919-video, 1,034-QA omni-modal benchmark for future event forecasting from audio-visual context, evaluates ~20 MLLMs, and proposes OFF, a LoRA instruction-tuning method on a 7K rationale-augmented dataset that improves open-source omni models.

Problem

Existing omni-modal benchmarks focus on retrospective understanding, and future forecasting benchmarks are text-only or vision-only, so MLLMs' ability to forecast future events from joint audio-visual context is unexplored.

Approach

Builds a future-forecasting benchmark via audio-coordinated video filtering, LLM-based causal pair mining with audio-contribution scoring, and four adversarial distractor types (visual-only, audio-only, delayed, reverse-causal), then uses rationale-augmented instruction tuning (OFF) to improve omni-modal forecasting and transfer to general benchmarks.</править_core_idea>

Key results

• Best proprietary model Gemini 3 Flash reaches 64.8% average accuracy on FutureOmni, while the best open-source omni model (Qwen3-Omni) reaches 53.05%.
• OFF fine-tuning improves video-SALMONN 2 from 46.03% to 49.90% (+3.87) and Qwen2.5-Omni from 47.48% to 48.51% on FutureOmni.
• OFF raises Qwen2.5-Omni on Video-MME from 53.77% to 55.51% and on WorldSense from 37.67% to 40.22%, showing transfer beyond forecasting.

Surprising findings

• Lack of knowledge accounts for only 2.5% of errors on Gemini 3 Flash, suggesting the gap stems from perception and reasoning rather than world knowledge.
• Omni-modal models score similarly on audio-only and video-only settings, which the authors interpret as evidence of balanced distractors.

Limitations

• Benchmark construction relies heavily on Gemini 2.5 Flash and GPT-4o automated steps, which may introduce model-specific biases.
• Attention analysis uses only LongVALE with a single model (Qwen2.5-Omni), limiting mechanistic generality.

Experimental setup

Datasets: FutureOmni FutureOmni-Train LongVALE WorldSense DailyOmni JointAVBench OmniVideoBench Video-MME MLVU

Benchmarks: FutureOmni WorldSense DailyOmni JointAVBench OmniVideoBench Video-MME MLVU

Metrics: accuracy

Baselines: Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Flash Claude Haiku 4.5 GPT-4o Qwen3-Omni Qwen2.5-Omni video-SALMONN 2 Ola MiniCPM-o 2.6 Qwen3-VL VideoLLaMA3

Builds on: LongVALE AVoCaDO LlamaFactory LoRA

Closest papers by meaning

• Rethinking Post-Training Recipes for Multimodal Time-Series Forecasting (NeurIPS 2026 · similarity 0.87)
• Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting (ICML 2025 · similarity 0.86)
• ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data (AAAI 2025 · similarity 0.86)
• SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation? (NeurIPS 2026 · similarity 0.86)
• Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers (CVPR 2025 · similarity 0.86)
• TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop (NeurIPS 2025 · similarity 0.86)