Research map / Vision-language and multimodal / Multimodal LLMs
Bridging Compressed Image Latents and Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a lightweight transform-neck plus a surrogate loss (CE with CLIP text embeddings + feature distillation) that maps compressed ELIC image latents into the intermediate layers of a CLIP visual encoder so frozen MLLMs can run on compressed images without decoding or back-propagating through the MLLM.
Problem
Image compression for cloud-hosted MLLMs: decoded images from human-perception codecs degrade MLLM task accuracy at low bitrates, while existing coding-for-machines methods require back-propagating task losses through billion-parameter MLLMs, which is infeasible.
Approach
A transform-neck (linear projection, one self-attention block, FFN, two layer norms) injects neural codec latents directly into the third layer of the CLIP ViT visual encoder, trained with a surrogate loss that only backpropagates through the partial visual encoder: a CLIP-text-based cross-entropy term plus an MSE feature distillation term, with a progressive schedule. Three scenarios cover a froz
Key results
- Compared with existing codecs, the method reports up to 60-80% bit-rate reduction at the same recognition accuracy (ELIC and VVC intra baselines).
- Decoding complexity is reduced by nearly 95% in kMAC/pixel versus reconstruction followed by post-processing (transform-neck 52.795 kMAC/pixel vs 1017.96 for post-processing pipeline), with 13.19M vs 64.16M parameters.
- The method is trained on one RTX 4090 with 24GB memory, approximately three days per rate point, whereas full-MLLM training is not feasible.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt ELIC latents with the transform-neck instead of feeding reconstructed images | MLLM-based vision tasks using ELIC · Captioning, VQA, REC, and few-shot classification | task performance | Helped |
| Use the post-processing network on reconstructed images | ELIC reconstruction baseline · MLLM-based vision tasks | task performance | Helped |
| Retrain the image codec jointly for human and machine perception (d2) | Transform-neck with ELIC · MLLM-based vision tasks | task performance | Helped |
| Retrain the encoder specifically for machine perception (d3) | Transform-neck with ELIC · MLLM-based vision tasks | task performance | Helped |
| Train with cross-entropy loss only (ablation) | Transform-neck (d1) · Image captioning | task performance | Hurt |
| Train with distillation loss only (ablation) | Transform-neck (d1) · Image captioning | task performance | Hurt |
| Combine cross-entropy and distillation losses with progressive training (ablation) | Transform-neck (d1) · Image captioning | rate-accuracy performance | Helped |
| Remove the first one or two CLIP visual encoder layers (ablation) | Transform-neck (d1) · Image captioning | task performance | No clear effect |
| Remove four or eight CLIP visual encoder layers (ablation) | Transform-neck (d1) · Image captioning | task performance | Hurt |
| Use the transform-neck with TIC instead of ELIC (ablation) | Transform-neck · Image captioning | task performance versus reconstruction baseline | Helped |
| Use a fixed human-perception codec and feed its reconstruction to the MLLM | ELIC reconstruction baseline · MLLM-based vision tasks | task performance | Hurt |
| Use the proposed method with a non-CLIP visual encoder | mPLUG-Owl2 and Osprey · MMBench and POPE | task performance versus reconstruction baseline | Helped |
Surprising findings
- Training with only the distillation loss performs far worse than the proposed combined surrogate loss, and cross-entropy alone also causes a significant drop.
- VVC intra coding performs worse than the ELIC-based reconstruction baseline across all tasks on 256x256 inputs.
Limitations
- Focused only on image compression; video or audio coding for MLLMs is left for future work.
- Results shown are for MLLMs sharing the CLIP ViT visual encoder in the main experiments; generalization to other encoders is only illustrated on two models.
- The few-shot classification task uses a self-designed 5-way 1-shot setting because the original code was unavailable.
Experimental setup
Datasets: ImageNet COCO SEED-Bench POPE
Benchmarks: COCO captioning VQA (Honeybee) REC (Shikra) few-shot classification (V2L-Tokenizer) MMBench (mPLUG-Owl2) POPE (Osprey)
Metrics: task accuracy vs bpp PSNR vs bpp kMAC/pixel parameter count
Baselines: Reconstruction (ELIC decoded images) Post-processing (U-Net) VVC intra coding (VTM 17.0) TIC
Builds on: ELIC TIC CLIP hyperprior-based neural image codec
Closest papers by meaning
- When MLLMs Meets Compression Distortion: A Coding Paradigm Tailored to MLLMs (ICLR 2026 · similarity 0.92)
- Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.89)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
- VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.88)
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.