Research map / Vision-language and multimodal / Cross-modal learning

OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OmniSelect, a training-free, modality-adaptive token pruning framework for Omni-LLMs that uses AudioCLIP relevance scores to choose audio-centric, video-centric, or uniform pruning and then prunes tokens within temporal groups.

Problem

Omni-modal LLMs process long audio-video token sequences, causing heavy compute and memory cost; existing training-free compression methods use fixed, modality-specific guidance that ignores that the relative importance of audio and video varies across queries.

Approach

Use AudioCLIP similarity between the query and per-group video/audio embeddings to dynamically select the pruning regime (uniform, video-centric, or audio-centric), allocate per-group pruning ratios via a sigmoid over similarity scores under a global budget, and prune audio by encoder attention (Top-K) and video by embedding redundancy (Bottom-K) within each temporal group.

Key results

• At 45% retained tokens on Qwen2.5-Omni-7B, VideoMME average 66.33 vs full-token 66.44, and DailyOmni average 60.65 vs OmniZip 59.98.
• On WorldSense with Qwen2.5-Omni-7B at 30% retained tokens and 128 frames, speedup of 1.33x in average latency (2.76s vs 3.66s) with accuracy 44.17 vs full 46.82.
• Retains 94%-99% of full-token accuracy across benchmarks while reducing GPU memory by roughly 2.6-2.8 GB.

Surprising findings

• OmniSelect at 45% retention exceeds full-token accuracy on several VideoMME long-video splits for Qwen2.5-Omni-3B (e.g., 53.3 vs 52.8 long).

Limitations

• Strategy selection uses manually designed threshold rules, and a single fixed threshold is suboptimal across samples.
• Relies on AudioCLIP, which has limited discriminative capability for fine-grained modality importance.
• Any-correct oracle results show a gap between selection and achievable accuracy, implying strategy selection errors.

Experimental setup

Datasets: WorldSense DailyOmni VideoMME OmniVideoBench

Benchmarks: WorldSense DailyOmni VideoMME OmniVideoBench

Metrics: accuracy GPU memory prefilling time average latency

Baselines: OmniZip DyCoke Random pruning

Builds on: OmniZip DyCoke AudioCLIP Q-Frame AKS

Closest papers by meaning

• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression (NeurIPS 2026 · similarity 0.91)
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.89)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.87)
• Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.87)
• A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.87)
• Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.87)