Research map / Video / Video-language understanding
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Oryx, a unified image/video/3D MLLM with a native-resolution vision encoder (OryxViT) and a dynamic compressor supporting 1x–16x visual token compression on demand.
Problem
Existing multimodal LLMs standardize visual inputs of varying resolution and duration to fixed-size images or patches, yielding similar token counts regardless of content, which is suboptimal for recognition precision and inefficient for long videos.
Approach
A SigLIP-based encoder (OryxViT) with interpolated position embeddings and variable-length flash attention that processes images at native resolution, plus a dynamic compressor that uses cross-attention-enhanced downsampling with a shared MLP projector to select compression ratios per input type.
Key results
- Oryx-1.5-32B reaches 67.3/74.9 on VideoMME (w/o and w/ subs) and 85.0 on NextQA, surpassing prior open-source SOTA on several video benchmarks.
- Oryx-1.5-32B scores 72.3 on MLVU and 86.3 on MMBench, and 92.7 on DocVQA.
- Oryx-7B reaches 58.3/62.6 on VideoMME and 68.6 on PerceptionTest; OryxViT native resolution is about 7% slower than SigLIP partitioning (146.5 vs 157.7 img/s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OryxViT at native resolution instead of dynamic partition (ablation) | Oryx visual encoder · DocVQA | score: 76.3 -> 78.5 | Helped |
| search for the optimal anchor resolution (ablation) | Oryx visual encoder · DocVQA | score: 78.5 -> 79.2 | Helped |
| replace the MLP connector with Dynamic Compressor (ablation) | Oryx architecture · VideoMME | score: 54.6 -> 55.4 | Helped |
| replace average pooling with DWConv in Dynamic Compressor (ablation) | Oryx Dynamic Compressor · VideoMME | score: 55.4 -> 55.0 | Hurt |
| replace average pooling with Conv-MLP in Dynamic Compressor (ablation) | Oryx Dynamic Compressor · MLVU | score: 59.3 -> 58.5 | Hurt |
| add long-form temporal training data (ablation) | Oryx · MLVU | score: 58.1 -> 59.3 | Helped |
| add 3D training data (ablation) | Oryx · ScanQA | METEOR: 11.7 -> 12.8 | Helped |
| add coarse correspondences to 3D training data (ablation) | Oryx · ScanQA | ROUGE-L: 32.7 -> 35.1 | Helped |
| use separate image and video MLPs instead of a shared MLP (ablation) | Oryx visual connector · MLVU | score: 59.3 -> 54.2 | Hurt |
| replace Dynamic Compressor with average pooling (ablation) | Oryx downsampling architecture · VideoMME | score: 55.4 -> 54.6 | Hurt |
| replace Dynamic Compressor with convolution (ablation) | Oryx downsampling architecture · MLVU | score: 59.3 -> 56.8 | Hurt |
| replace Dynamic Compressor with Q-former (ablation) | Oryx downsampling architecture · MLVU | score: 59.3 -> 35.3 | Hurt |
Surprising findings
- Native-resolution SigLIP without adaptation collapses (DocVQA 17.1, MMBench 15.8), so the encoder must be pre-trained for native resolution.
- Parameter-free average pooling outperforms learned downsampling (DWConv, Conv-MLP) and Q-former in the compressor.
Limitations
- Uniform frame sampling loses temporal context; the model struggles with time perception.
- Lacks domain knowledge (e.g., chemistry), causing knowledge-reasoning failures.
- Relies on open-source data with GPT-4o-generated needle-in-a-haystack data, and the training pipeline is not fully scaled.
Experimental setup
Datasets: LLaVA-1.5 558k captions LLaVA-NeXT Cauldron Cambrian-1 VideoChatGPT-Plus ShareGPT4Video LLaVA-Hound CinePile NextQA PerceptionTest MovieNet ScanQA
Benchmarks: VideoMME NextQA MVBench PerceptionTest MMBench-Video VideoChatGPT VDC MLVU LongVideoBench MMBench MMMU DocVQA
Metrics: accuracy CIDEr METEOR ROUGE-L BLEU GPT score throughput memory
Baselines: LLaVA-OneVision LLaVA-Next-Video VILA-1.5 VideoLLaMA2 LongVA Kangaroo 3D-LLM Cambrian-1
Builds on: SigLIP LLaVA NaViT FlashAttention
Closest papers by meaning
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.87)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.86)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.86)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.86)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.