Research map / Vision-language and multimodal / Human and embodied modeling
VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents UniViTAR, a family of 0.3B–1B vision transformers that process images and videos at native resolution, trained with a four-stage recipe mixing sigmoid contrastive learning, DINOv2 feature distillation, resolution curriculum, and image-video alternation on public data.
Problem
Standard ViTs force uniform square low-resolution inputs, discarding native aspect ratios and spatial detail; existing native-resolution models lack a systematic training recipe and a unified image/video backbone family.
Approach
Upgrade the ViT with 2D-RoPE, SwiGLU, RMSNorm, QK-Norm and LayerScale; train with a staged recipe (fixed-res contrastive+distillation pretraining, then native-resolution tuning with packed variable-length sequences, then inter-batch image/video alternation), with distillation from DINOv2-g decayed to zero.
Key results
- UniViTAR-1B reaches 83.5% average zero-shot ImageNet-variant accuracy, above SigLIP-SO400M at 384 (81.7%) with less training data.
- UniViTAR-1B achieves 86.0 average linear/attentive probing accuracy on ImageNet variants, and 45.4 mIoU linear segmentation on ADE20K, +6.1 over OpenCLIP-G.
- Average zero-shot video classification rises from 68.0 (0.3B) to 69.0 (1B) across K400, UCF101 and HMDB51.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| alternate image-only and video-only batches instead of mixing both modalities within batches (ablation) | UniViTAR-0.3B · ImageNet-1K | zero-shot accuracy: 70.46 -> 71.25 | Helped |
| alternate image-only and video-only batches instead of mixing both modalities within batches (ablation) | UniViTAR-0.3B · ImageNet-A | zero-shot accuracy: 45.89 -> 48.60 | Helped |
| alternate image-only and video-only batches instead of mixing both modalities within batches (ablation) | UniViTAR-0.3B · K400 | zero-shot accuracy: 58.82 -> 61.01 | Helped |
| alternate image-only and video-only batches instead of mixing both modalities within batches (ablation) | UniViTAR-0.3B · UCF101 | zero-shot accuracy: 75.15 -> 77.66 | Helped |
| increase training data scale from Merged-100M to Merged-1B at the same seen-sample count (ablation) | UniViTAR-0.3B · zero-shot image classification | overall accuracy: 60.8 -> 64.2 | Helped |
| increase training data scale from Merged-100M to Merged-1B at the same seen-sample count (ablation) | UniViTAR-0.3B · zero-shot image retrieval | overall retrieval: 67.0 -> 68.9 | Helped |
| preserve image aspect ratio rather than use fixed-resolution center crops (ablation) | UniViTAR-0.6B · ImageNet-1K and ImageNet-A | zero-shot accuracy | Helped |
| increase sequence length under fixed-resolution or native-aspect-ratio inference (ablation) | UniViTAR-0.6B · ImageNet-1K and ImageNet-A | zero-shot accuracy | Mixed |
| progress from fixed-resolution training through native-resolution training to image-video training (ablation) | UniViTAR · zero-shot image classification | accuracy | No clear effect |
| add dynamic-resolution training in Stage 3 (ablation) | UniViTAR · zero-shot video classification | accuracy | Helped |
| add image-video cross-modal training in Stage 4 (ablation) | UniViTAR · zero-shot video classification | accuracy | Helped |
| increase the maximum video sequence-length limit (ablation) | UniViTAR-0.6B · K400 | zero-shot accuracy | Helped |
Surprising findings
- Models pretrained at fixed 224 resolution generalize to variable-length native-resolution inputs after limited native-resolution tuning.
- Alternating image and video batches outperforms mixing them within a batch on image and video zero-shot metrics.
Limitations
- Video performance lags behind models trained exclusively on video-caption data (e.g., InternVideo2-6B), which the authors attribute to image data mixing.
- Dense-prediction and MLLM evaluations use frozen backbones with limited resolution settings, and several multimodal gains over baselines are small (e.g., average 49.6 vs 48.5 for SigLIP-SO400M).
Experimental setup
Datasets: DataComp-1B COYO LAION-2B DFN-2B LAION-400M CC12M CC3M Panda-70M WebVid-10M InternVid-10M-FLT LLaVA-CC3M-Pretrain ALLaVA-Caption
Benchmarks: ImageNet-1K ImageNet-A ImageNet-R ImageNet-V2 ImageNet-Sketch ObjectNet Flickr30K MS-COCO K400 UCF101 HMDB51 ActivityNet
Metrics: top-1 accuracy recall@1 mIoU average score
Baselines: CLIP-L SigLIP-L SigLIP-SO400M OpenCLIP-G InternViT-6B EVA-8B DFN-H AIMv2-H ViCLIP-L UMT-L
Builds on: NaViT SigLIP CLIP DINOv2 InternVideo
Closest papers by meaning
- Native-Resolution Image Synthesis (NeurIPS 2025 · similarity 0.88)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.88)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.86)
- InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD (NeurIPS 2024 · similarity 0.86)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.86)
- Bootstrapping SparseFormers from Vision Foundation Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.