Research map / Video / Video-language understanding
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OneVision-Encoder, a ViT that selects codec-derived motion/residual patches from video (HEVC-style Codec Patchification) and is trained with a cluster-discrimination objective over a million-scale concept bank, evaluated via LMM probing and attentive probing.
Problem
Video encoders process dense pixel grids uniformly, spending capacity on redundant static regions; existing frame-centric and contrastive pretraining methods lack structured, sparsity-aware visual representations.
Approach
Use codec-exposed motion vectors and residual energy from HEVC P-frames to select a fixed budget of salient patches (3.1%-25%) while keeping I-frames and dense temporal coverage, combined with 3D-RoPE and a unified object- and motion-level cluster discrimination loss.
Key results
- Under identical multimodal fine-tuning, OneVision-Encoder outperforms SigLIP2 across 16 video, image, and document benchmarks with a Qwen3-4B LMM.
- Under attentive probing, OV-Encoder achieves 12.7% absolute Top-1 gain over SigLIP2 on Diving48 at identical patch budget (abstract cites 17.1% over SigLIP2 and 8.1% over DINOv3).
- Codec patch selection reduces patch processing by 75.0%-96.9% versus dense 64-frame processing (16,384 patches) while retaining competitive accuracy.
Surprising findings
- Pretrained on about 100B caption tokens, the encoder outperforms Qwen3-ViT trained on more than 2.1T tokens in the controlled LMM probe.
- Replacing codec-selected motion patches with unrelated-video motion patches degrades performance more than replacing them with non-motion patches.
Limitations
- Center bias in codec-selected patches leads to weaker peripheral coverage without chunk-wise sampling.
- Codec-based selection relies on HEVC-encoded signals (motion vectors and residuals) being available from the bitstream.
Experimental setup
Datasets: LAION-400M COYO-700M OBELICS Zero250M HowTo100M Panda-70M Kinetics-710 Something-Something-V2 LLaVA-Next LLaVA-Next-Videos LLaVA-OneVision-1.5
Benchmarks: ChartQA DocVQA InfoVQA AI2D MMBench-EN OCRBench OCRBench v2 MMStar RealWorldQA MVBench MLVU-dev NExT-QA
Metrics: accuracy Top-1 accuracy
Baselines: SigLIP2 Qwen3-ViT DINOv3 MetaCLIP2 AIMv2
Builds on: MLCD SigLIP 3D RoPE LLaVA-OneVision-1.5 HEVC
Closest papers by meaning
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.87)
- HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.87)
- Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation (NeurIPS 2026 · similarity 0.87)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.86)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.86)
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.