Research map / Vision-language and multimodal / Cross-modal learning
A Mixed Diet Makes DINO An Omnivorous Vision Encoder
Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson, Ye Xia, Skanda Koppula, Andre Araujo, Joao Carreira, Niloy J. Mitra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-train with cross-modal alignment plus anchoring (Omnivorous) | DINOv2 ViT-B/14 · ScanNet cross-modal retrieval (GAP) | R@1: 4.6 -> 46.1 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · MOVi cross-modal retrieval | R@1: 15.5 -> 86.2 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · ImageNet linear probe | accuracy: 0.801 -> 0.835 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · NYUv2 depth (linear head) | RMSE: 0.405 -> 0.377 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · NYUv2 depth (DPT head) | RMSE: 0.297 -> 0.297 | No clear effect |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · iNaturalist k-NN | accuracy: 78.53 -> 77.49 | Hurt |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · RP2K k-NN | accuracy: 66.83 -> 70.48 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · Zero-shot cross-modal depth, Seg input | RMSE: 1.536 -> 0.532 | Improved |
| post-train with Omnivorous alignment | DINOv2 ViT-B/14 · Zero-shot cross-modal depth, NOCS input | RMSE: 1.996 -> 1.075 | Improved |
| add modality mixup (alpha_max 0 -> 0.5) (ablation) | Omnivorous DINOv2 · Cityscapes segmentation (linear) | mIoU: 0.624 -> 0.632 | Improved |
| add modality mixup (alpha_max 0 -> 1.0) (ablation) | Omnivorous DINOv2 · NYUv2 depth delta1 | delta1: 0.899 -> 0.891 | Hurt |
| train adapter on top instead of fine-tuning final blocks (ablation) | Omnivorous DINOv2 · NAVI 3D correspondence | PCK@0.0: 29.00 -> 28.15 | Hurt |
Closest papers by meaning
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.87)
- Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation (NeurIPS 2026 · similarity 0.87)
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.86)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.86)
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.85)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
