Research map / Large language models / Agents, code and math
Rethinking Visual Intelligence: Insights from Video Pretraining
Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a pretrained video diffusion model instead of an LLM as the backbone | LoRA-adapted CogVideoX1.5-5B vs Qwen3-4B-Instruct-2507 · Route planning (Shortest Path, Maze) | training samples needed: up to a tenfold reduction in data requirements in low-sample regimes | Improved |
| use a video diffusion backbone instead of an LLM | CogVideoX1.5-5B · Visual games (Hitori, Sudoku, Sudoku Mini, Connect 4) | accuracy vs training set size | Improved |
| use a video diffusion backbone instead of an LLM | CogVideoX1.5-5B · Chess Mate-in-1 | accuracy vs training set size | Hurt |
| train on 13x13 mazes and test on 21x21 with a video diffusion backbone | CogVideoX1.5-5B · Maze Generalization | accuracy | Improved |
| use a video diffusion backbone instead of an LLM | CogVideoX1.5-5B · 1D elementary cellular automata (delta 0.9) | examples to reach 0.9 accuracy | No clear effect |
| use a video diffusion backbone instead of an LLM | CogVideoX1.5-5B · Life-like cellular automata | examples to reach 0.9 accuracy | Improved |
| use a video diffusion backbone, with longer prediction horizon | CogVideoX1.5-5B · Langton's Ant (horizon 2, 3, 5, 10) | accuracy vs training set size | Improved |
| raise LoRA rank from 32 to 64 for Qwen3 (and lower 64 to 32 for CogVideoX) (ablation) | Qwen3 and CogVideoX1.5-5B LoRA · ARC and structured tasks | accuracy: rank 32 vs rank 64 | No clear effect |
Closest papers by meaning
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.88)
- Pathways of Visual Information Flow in Vision-Language Models (NeurIPS 2026 · similarity 0.88)
- Exploring Visual Pretraining for Learning Language Intelligence (CVPR 2026 · similarity 0.88)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
