Research map / Vision-language and multimodal / Multimodal LLMs

Omni-Interactive Universal Embedder

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OmniUE, an omni-LLM embedder that adds SAM-3 and SAM-Audio segmenter streams for visual/audio interaction prompts plus learnable multi-layer token aggregation, and introduces OmniCHOIR, a text-video-audio-to-audio retrieval benchmark with mask and span interactions.

Problem

Existing LLM-based multimodal embedders accept mainly text instructions (and at most image visual prompts), so they cannot take visual region or audio span interactions across video and audio, and no benchmark evaluates omni-interactive embedding.

Approach

Combine an omni-LLM (Qwen2.5-Omni) with frozen SAM-3 and SAM-Audio segmenters whose pre-decoder features are projected into the LLM as interaction tokens, and aggregate hidden states of several appended learnable tokens across layers with learned layer-token softmax weights to form the embedding.

Key results

• OmniUE-7B reaches 51.8 overall on MMEB-v2-video, +4.3 over the best 7B baseline, and OmniUE-3B reaches 48.4 (+5.1 over 2B/3B baselines).
• OmniUE-7B scores 55.2 overall on SCaR versus 27.8 for the best 7B baseline (VIRTUE-7B), a +27.4 gain.
• On OmniCHOIR, OmniUE-7B reaches 28.4 Recall@1 with text+span+mask versus 22.3 for LCO-Emb-7B under text only.

Surprising findings

• Removing the visual and audio segmenter streams at inference time still improves over baselines on MMEB-v2-video, suggesting the segmenter connectors help even without explicit prompts.
• Cropping audio spans or converting spans to text does not match the gains from the span-based omni-interaction representation.

Limitations

• Depends on pretrained components (Qwen2.5-Omni, SAM-3, SAM-Audio), so it cannot be trained from scratch.
• Masks trade off interaction precision against latency (about 9.4s vs 1.3s per sample).
• OmniCHOIR is built from 479 instances, a relatively small benchmark, and its negatives are synthesized with an LLM.

Experimental setup

Datasets: AudioSet WavCaps MSR-VTT PE-Videos VALOR VATEX AudioCaps Clotho Panda70m Shot2Story VGGSound Youtube8m

Benchmarks: MMEB-v2-video MAEB SCaR OmniCHOIR VGGSound VALOR-32K

Metrics: Recall@1 accuracy average score per meta-task

Baselines: VLM2Vec-v2 UME-R1 UniME-V2 CAFe Omni-Embed-Nemotron LCO-Emb e5-omni CLAP Qwen2-Audio-7B ImageBind WAVE-7B VIRTUE

Builds on: Qwen2.5-Omni SAM-3 SAM-Audio LCO-Emb VLM2Vec GradCache

Closest papers by meaning

• OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces (ICLR 2025 · similarity 0.91)
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.90)
• OmniBench: Towards The Future of Universal Omni-Language Models (NeurIPS 2025 · similarity 0.90)
• Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.88)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.88)