Research map / Vision-language and multimodal / Cross-modal learning
OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data
Yan Zhao, Zhengxue Cheng, Junxuan Zhang, Dajiang Zhou, Qunshan Gu, Qi Wang, Li Song
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add modality-routing contextual learning and feedforward MoE on top of unified tokenizer (ablation) | OmniZip-S · Spider | bits/Byte: 1.662 -> 1.320 | Improved |
| add modality-routing contextual learning (feedforward routing already present) (ablation) | OmniZip-S · enwik9 | bits/Byte: 1.424 -> 1.370 | Improved |
| add modality-routing feedforward (contextual routing already present) (ablation) | OmniZip-S · enwik9 | bits/Byte: 1.410 -> 1.370 | Improved |
| add reparameterization training (ablation) | OmniZip-S · enwik9 | bits/Byte: 1.390 -> 1.370 | Improved |
| apply contextual MoE routing to Value layer instead of Key layer (ablation) | OmniZip-S · Kodak | bits/Byte: 3.614 -> 3.307 | Improved |
| apply contextual MoE routing to Value instead of Receptance (ablation) | OmniZip-S · enwik9 | bits/Byte: 1.311 -> 1.370 on enwik9; 4.281 -> 4.155 on LibriSpeech | Mixed |
| extend contextual routing to more projections (R,K,V) (ablation) | OmniZip-S · multi-modal (7 representative datasets) | bits/Byte: Kodak 3.307 -> 3.358; LibriSpeech 4.155 -> 4.291 | No clear effect |
| increase feedforward expert hidden factor from 1x to 2x (ablation) | OmniZip-S · Kodak | bits/Byte: 3.502 -> 3.307 | Improved |
| increase hidden factor from 2x to 4x (ablation) | OmniZip-S · Kodak | bits/Byte: 3.307 -> 3.316 | No clear effect |
| use RWKV-7 backbone instead of Transformer (ablation) | lightweight text compressor (3.2M params) · enwik9 | bits/Byte: 1.984 -> 1.658 | Improved |
| route all blocks instead of one block (4 experts, top-k 2) (ablation) | OmniZip-S · Kodak | bits/Byte: 3.358 -> 3.307 | Improved |
| increase model size from OmniZip-S to OmniZip-L | OmniZip · enwik9 | bits/Byte: 1.370 -> 0.980 | Improved |
Closest papers by meaning
- OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data (ICML 2025 · similarity 0.89)
- Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.86)
- MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.86)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.86)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
