Research map / Vision-language and multimodal / Multimodal LLMs

When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how compression distortion affects different feature levels in MLLM vision encoders, then proposes CoTAM, a CLIP-guided learned image codec with a multi-level decoder adapter that saves up to 35.99% bitrate on MLLM tasks.

Problem

Conventional human-centric and coding-for-machine image codecs are not suited to MLLMs, which need both low-level details and high-level semantics; compression degrades MLLM performance unevenly across tasks.

Approach

Uses shallow CLIP [CLS] attention to build a quantized importance map for bit allocation in a learned codec, and a decoder with a reconstruction prior, a single-transformer-block latent adapter, and a multi-level (low- and high-level) fidelity loss, with hierarchical guidance for high-resolution inputs.

Key results

• Achieves up to 35.99% bitrate saving at the same MLLM performance on LLaVA-1.5-7B with ELIC as base codec, outperforming prior neural codecs.
• Integrating with DCAE yields similar performance gains, per the paper.
• Shows improvements on LLaVA-1.5-13B, LLaVA-OneVision-7B (high-res), InternVL2-8B, and Video-MME.

Surprising findings

• Directly fine-tuning the codec with the MLLM frozen causes training collapse and loss of comprehension abilities (ELIC-CFT).
• Compression-induced similarity drop is concentrated in the early phase of the deep layers rather than uniformly across depth.

Limitations

• Intra-frame only; video handled frame-by-frame without modeling temporal redundancy.
• Analysis isolates the vision encoder and does not study LLM-side effects.
• Freezing the codec trades off against reconstruction quality (slight PSNR degradation).

Experimental setup

Datasets: CC3M LLaVA-Instruct

Benchmarks: MME MMBench TextVQA POPE SeedBench VQAv2 MMMU Video-MME

Metrics: bitrate saving (BD-rate) task accuracy/score PSNR

Baselines: ELIC DCAE Bridge-d1 Bridge-d3 ICMH-adapt

Builds on: ELIC DCAE CLIP multi-quantizer variable-rate compression

Closest papers by meaning

• Bridging Compressed Image Latents and Multimodal Large Language Models (ICLR 2025 · similarity 0.92)
• Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.90)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.89)
• Towards Unified Human Perception and Machine Understanding: Token Flow Guided Compression Framework (CVPR 2026 · similarity 0.89)
• Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.89)
• LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)