Research map / Vision-language and multimodal / CLIP and zero-shot
ViT-Lens: Towards Omni-modal Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViT-Lens, which trains small modality-specific lenses that feed novel modalities (3D, depth, audio, tactile, EEG) into a frozen pretrained ViT, aligned contrastively to CLIP anchors, and plugs the lenses into MFMs for zero-shot any-modality use.
Problem
Omni-modal representation learning for rare modalities is data-limited and costly, so models generalize poorly to novel categories in modalities beyond image, video and text.
Approach
Reuse a frozen pretrained ViT as a shared encoder for new modalities by training only a modality embedding module and a lens that maps modality tokens into the ViT input space, aligned contrastively to a frozen CLIP anchor space; the resulting lenses can be plugged into multimodal foundation models without extra instruction tuning.
Key results
- Zero-shot 3D classification on Objaverse-LVIS without LVIS training reaches 50.1% top-1, surpassing OpenShape-PointBERT trained on the full set by 3.3%.
- On AudioSet linear/classification, VIT-LENS-L (I+T) reaches 26.7 mAP vs AudioCLIP 25.9 mAP.
- On ImageNet-EEG, VIT-LENS-L (I+T) achieves 42.7% test accuracy vs 19.2% for DreamDiffusion-L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIT-LENS-G instead of OpenShape-PointBERT, trained on OpenShape-Triplets | 3D zero-shot classification · Objaverse-LVIS | Top1 accuracy (%): 46.8 -> 52.0 | Helped |
| Train VIT-LENS-G without Objaverse-LVIS shapes | 3D zero-shot classification · Objaverse-LVIS | Top1 accuracy (%): 50.1 | Helped |
| Align audio to images and text rather than images alone | VIT-LENS-L · Audioset | mAP: 23.1 -> 26.7 | Helped |
| Use image and text as anchors instead of image alone | VIT-LENS-L depth encoder · NYU-D | accuracy: 64.2 -> 68.5 | Helped |
| Use image and text as anchors instead of image alone | VIT-LENS-L depth encoder · SUN-D | accuracy: 37.4 -> 52.2 | Helped |
| Initialize S-Attn Lens with pretrained weights rather than random initialization (ablation) | VIT-LENS-B · ModelNet40 | Top1 accuracy: 63.8 -> 65.4 | Helped |
| Initialize S-Attn Lens with pretrained weights rather than random initialization (ablation) | VIT-LENS-B · SUN-D | accuracy: 48.6 -> 50.9 | Helped |
| Use Iter-CS-Attn instead of pretrained-weight S-Attn Lens (ablation) | VIT-LENS-B · ModelNet40 | Top1 accuracy: 65.4 -> 65.4 | No clear effect |
| Use Iter-CS-Attn instead of pretrained-weight S-Attn Lens (ablation) | VIT-LENS-B · SUN-D | accuracy: 50.9 -> 47.5 | Hurt |
| Train ModEmbed and ViT from scratch instead of using the VIT-LENS architecture (ablation) | VIT-LENS-B · ImageNet-EEG | accuracy: 20.5 vs 35.9 | Hurt |
| Freeze pretrained ViT and omit the Lens, using only ModEmbed + ViT (ablation) | VIT-LENS-B · SUN-D | accuracy: 36.8 vs 50.9 | Hurt |
| Fine-tune the full pretrained ViT rather than use VIT-LENS (ablation) | Modality encoder · Touch-and-Go Material | accuracy: 59.4 vs 63.6 | Hurt |
Surprising findings
- Simply adding the modality embedding to a frozen pretrained ViT performs poorly (e.g., 24.8% on TAG-M), so the lens is needed to make the frozen ViT effective.
Limitations
- May inherit biases and errors from the pretrained ViT.
- Not readily usable for real-world deployment as stated by the authors.
- Improvements on some modalities are modest relative to the baselines, and some benchmarks use larger pretraining data (gray-out entries).
Experimental setup
Datasets: ShapeNet 3D-FUTURE ABO Objaverse ULIP-ShapeNet ULIP-2-Objaverse OpenShape-Triplets SUN RGB-D AudioSet Touch-and-Go ImageNet-EEG
Benchmarks: ModelNet40 Objaverse-LVIS ScanObjectNN SUN RGB-D (depth-only) NYU-v2 (depth-only) AudioSet ESC-50 Clotho AudioCaps VGGSound MSR-VTT Touch-and-Go
Metrics: accuracy mAP recall@k top-1 accuracy
Baselines: ImageBind ULIP OpenShape PointCLIP v2 AudioCLIP CMC MultiMAE DreamDiffusion
Builds on: CLIP ULIP OpenShape ImageBind Perceiver-style cross-attention
Closest papers by meaning
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.88)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.86)
- LoMo: Local Modality Substitution for Deeper Vision-Language Fusion (NeurIPS 2026 · similarity 0.86)
- LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning (NeurIPS 2026 · similarity 0.86)
- OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.86)
- XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.