Research map / Vision-language and multimodal / Cross-modal learning
OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Binds 14 pre-trained unimodal-pair specialist embedding spaces to EVA-CLIP-18B via pseudo-paired retrieval and learns MoE-style router weights, yielding 7B-30B omni (3D, audio, image, text) representation models.
Problem
Large-scale omni multimodal representation across 3D, audio, image, and language is hard to train from scratch due to scarce paired data across all modalities, and naive fixed-weight averaging of pre-trained spaces causes knowledge interference and trade-offs.
Approach
Bind many existing specialist spaces onto a common anchor (EVA-CLIP-18B) using lightweight projectors trained on item-level pseudo pairs retrieved across modalities, then combine encoder outputs per modality with learned softmax routers trained by a cross-modal overall alignment loss and a language-decoupling loss.
Key results
- OmniBind-Full reaches 46.55 R@1 on Objaverse 3D-image retrieval versus 2.54 for Ex-MCR and 5.86 for PointBind.
- OmniBind-Large achieves 25.57 mAP on AudioSet zero-shot, above the best prior baseline ImageBind++ at 19.69.
- Ablation shows L_align plus L_dec routing gives the best overall results over manually weighted averaging variants, e.g., COCO R@1 62.64 vs 61.38 for the best manual setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the number of bound spaces and model scale from OmniBind-Base to OmniBind-Full (ablation) | OmniBind · AudioSet | mAP: 21.19 -> 25.14 | Helped |
| add L_dec to a router trained with L_align (ablation) | OmniBind-Full router · COCO | R@1: 59.43 -> 62.64 | Helped |
| add L_dec to a router trained with L_align (ablation) | OmniBind-Full router · AudioCaps | R@1: 43.71 -> 46.72 | Helped |
| add L_align to a router trained with L_dec (ablation) | OmniBind-Full router · AudioCaps | R@1: 48.48 -> 46.72 | Hurt |
| add L_align to a router trained with L_dec (ablation) | OmniBind-Full router · COCO | R@1: 61.86 -> 62.64 | Helped |
| add L_dec to the router (ablation) | OmniBind-Full · language embedding pairs from AudioCaps and COCO captions | cosine similarity: 0.0828 to 0.0517 | Helped |
| use fixed manual weights instead of input-dependent weight routing (ablation) | OmniBind · AudioCaps and COCO | R@1: AT high: 44.47; VT high: 33.24; PVT high: 34.95; Mean: 40.31; routed: 46.72 | Mixed |
| add L_align to the router (ablation) | OmniBind-Full · Objaverse-LVIS 3D classification | Top1: 52.93 -> 53.56 | Helped |
Surprising findings
- Adding the language decoupling loss reduced average cosine similarity between language embeddings from 0.0828 to 0.0517.
- Audio-image and 3D-image alignment improved beyond source spaces, suggesting cross-space knowledge transfer via the image anchor.
Limitations
- Only 14 existing spaces and four modalities are integrated; saturation of binding-based scaling is unexplored.
- Only three basic applications are demonstrated.
- Performance is bounded by the source specialist spaces; OmniBind-Full roughly matches the individual best rather than exceeding it on most tasks.
Experimental setup
Datasets: Objaverse AudioSet ImageNet AudioCaps Clotho COCO Flickr30K VGG-SS FlickrNet ESC-50 ScanObjectNN ModelNet40
Benchmarks: AudioSet ESC-50 ImageNet-1K Objaverse-LVIS ScanObjNN ModelNet40 AudioCaps Clotho VGG-SS FlickrNet COCO Flickr-30K
Metrics: zero-shot top-1 accuracy zero-shot top-5 accuracy mAP recall@1 recall@5
Baselines: Uni3D C-MCR LanguageBind ImageBind ImageBind++ InternVL-IB++ Ex-MCR PointBind
Builds on: FreeBind ImageBind C-MCR Ex-MCR Uni3D
Closest papers by meaning
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.91)
- OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.89)
- FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion (ICML 2024 · similarity 0.89)
- Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.87)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.87)
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.