Research map / Vision-language and multimodal / Cross-modal learning
OmniFit: Bridging Modalities via Layer-Adaptive Token Compression for Omnimodal Large Language Models
Zining Wang, Zhihang Yuan, Yingjie Zhai, Wenshuo Li, Han Shu, Ruihao Gong, Jinyang Guo, Xianglong Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LAHP to RandomDrop (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 50.1 -> 55.3 | Improved |
| add LAHP to ARTS (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 55.4 -> 62.0 | Improved |
| add modality preference profiling on top of ARTS w/ TRP (ablation) | OmniFit on Qwen2.5-Omni-3B · WorldSense, retain 20% | accuracy: 36.5 -> 45.1 | Improved |
| add token redundancy profiling to ARTS (ablation) | OmniFit on Qwen2.5-Omni-3B · WorldSense, retain 20% | accuracy: 28.6 -> 36.5 | Improved |
| replace RandomDrop with ARTS selection (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 50.1 -> 55.4 | Improved |
| use cumulative product schedule instead of linear decay (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 59.8 -> 62.0 | Improved |
| use cumulative product schedule instead of cosine decay (ablation) | OmniFit on Qwen2.5-Omni-3B · MLVU, retain 20% | accuracy: 65.1 -> 67.2 | Improved |
| increase calibration samples from 16 to 256 (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 58.2 -> 61.9 | Improved |
| increase calibration samples from 256 to 2048 (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 61.9 -> 62.0 | No clear effect |
| change calibration dataset from AVQA+Ola to other sources (ablation) | OmniFit on Qwen2.5-Omni-3B · Daily-Omni | accuracy: 59.8 to 60.3 | No clear effect |
| vary DPC-KNN K and M (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 61.4 to 62.0 | No clear effect |
| use pruning instead of merging for dropped tokens (ablation) | OmniFit on Qwen2.5-Omni-3B · VideoMME, retain 20% | accuracy: 62.0 -> 60.5 | Hurt |
Closest papers by meaning
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models (NeurIPS 2026 · similarity 0.92)
- OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts (CVPR 2025 · similarity 0.89)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.88)
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.88)
- Omni-Interactive Universal Embedder (NeurIPS 2026 · similarity 0.87)
- When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
