Research map / Video / Video-language understanding
OpenCoF: Learning to Reason Through Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 17K-video, 11-task video reasoning dataset (OpenCoF-17K), fine-tunes Wan2.2-I2V-A14B with LoRA into Wan-CoF, and adds learnable visual or textual reasoning tokens to test chain-of-frame reasoning gains on four external benchmarks.
Problem
Video generation models have weak reasoning ability (temporal coherence, physical/spatial consistency, logic) and little dedicated training for chain-of-frame reasoning; prior work mostly built benchmarks or domain-specific training.
Approach
Curate a diverse four-source reasoning video dataset and show data-only LoRA fine-tuning transfers to external benchmarks, then add learnable Visual Reasoning Tokens inside the DiT visual sequence and Textual Reasoning Tokens prepended to the text condition, analyzed via attention.
Key results
- Wan-CoF raises MME-CoF overall from 1.00 to 1.30 and Gen-ViRe average from 0.304 to 0.391 over the Wan2.2-I2V-A14B baseline.
- Wan-CoF(vt) reaches 0.441 on Gen-ViRe and 59.6 on RULER-Bench; Wan-CoF(tt) reaches 1.35 on MME-CoF and 8.8 POC on VIPER.
- VIPER POC improves from 3.3 (baseline) to 7.5 with data-only fine-tuning, a +4.2 gain.
Surprising findings
- Doubling reasoning tokens from 16 to 32 does not uniformly help; n=16 outperforms n=32 on three of four benchmarks for both variants.
- RULER-Bench overall gain from data-only fine-tuning is small (+1.0), with Visual Consistency dropping by 4.0 on Vision split.
Limitations
- vt and tt are investigated separately; combining them is left to future work.
- Open-source models still score near zero on several sub-dimensions; gains are small on some benchmarks (e.g., RULER-Bench +1.0).
Experimental setup
Datasets: OpenCoF-17K Mate in One (Chess) Sudoku-Easy Geo170K VR-Bench VBVR BridgeData V2
Benchmarks: MME-CoF Gen-ViRe VIPER RULER-Bench
Metrics: overall score (0-4 scale for MME-CoF) average score (Gen-ViRe, 0-1) POC@1.0 (VIPER) overall score 0-100 (RULER-Bench)
Baselines: Wan2.2-I2V-A14B Wan2.2-TI2V-5B HunyuanVideo-I2V Hunyuan-1.5 Kling-v1 Seedance-1.0-Pro Seedance-1.5 Veo-3.1 Sora-2
Builds on: Wan2.2 LoRA Chain-of-Thought
Closest papers by meaning
- CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models (NeurIPS 2026 · similarity 0.89)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.89)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.89)
- VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.