Research map / Vision-language and multimodal / Multimodal LLMs

Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BlackSwanSuite, a video benchmark of 1,655 surprising-event clips (pre/main/post splits) testing abductive and defeasible reasoning of VLMs via Forecaster, Detective, and Reporter tasks in generative, MCQ, and Y/N variants.

Problem

Commonsense abductive and defeasible reasoning of vision-language models is poorly evaluated, since existing video benchmarks focus on typical, predictable events rather than unexpected ones where models cannot rely on statistical recall.

Approach

Build video benchmarks around expectation-violating events by splitting each clip into pre-event, main-event and post-event segments, hiding parts of the video to test abduction (explaining hidden events) and revising hypotheses when new visual evidence appears (defeasibility).

Key results

• GPT-4o reaches 65.1% on Detective-MCQ and 62.4% on Detective-Y/N versus human 90.0% and 85.3%.
• On Reporter-Y/N, GPT-4o scores 60.1% against human 92.0%, a gap of about 32 points.
• Adding human-written perception and comprehension to LLaVA-Video raises Detective-MCQ accuracy on a 150-question subset from 58.6% to 68.6%.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought improves Detective-MCQ for GPT-4o (+5.3) but hurts Reporter-MCQ (-3.6).
• Larger LLaVA-Video-72B only modestly beats the 7B variant on Detective Y/N (56.93 vs 59.3) and is worse there.

Limitations

• Human-written annotations come from a single annotator per video, with some annotation mistakes noted by experts.
• Videos are short (median 8.83 s) and drawn from the Oops! YouTube fail dataset, limiting diversity.
• Generative evaluation relies on CLIP and Llama 3.1 8B judgments that the authors themselves find unreliable for open-ended answers.

Experimental setup

Datasets: Oops! dataset

Benchmarks: BlackSwanSuite

Metrics: accuracy CLIP score LLM-Match human rating (correctness, depth, detail, visual entailment) BLEU ROUGE-L

Baselines: GPT-4o Gemini 1.5 Pro VideoChat2 VideoLLaMA 2 VILA-1.5 LLaVA-Video

Builds on: Oops! dataset CLIP score LLaVA-Video

Closest papers by meaning

• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.91)
• AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs (AAAI 2026 · similarity 0.90)
• Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.90)
• Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
• MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.89)
• MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.89)