Research map / Vision-language and multimodal / Multimodal LLMs

Efficient Multi-modal Long Context Learning for Training-free Adaptation

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EMLoC, a training-free method that compresses many-shot multimodal demonstration contexts into compact KV memory via chunk-wise compression and layer-wise adaptive pruning gated by a JS-divergence check.

Problem

Multi-modal long-context in-context learning with many demonstrations gives strong adaptation but incurs prohibitive inference compute and memory costs, while fine-tuning requires parameter updates.

Approach

Process demonstrations in chunks, and for each chunk prune KV tokens layer by layer (top to bottom), choosing the smallest retention ratio per layer whose output distribution stays within a Jensen-Shannon divergence threshold of the unpruned output, using answer-token attention as importance.

Key results

• On ImageNet100 with 200 examples, EMLoC reaches 63.7% accuracy at 22.4% of the context (3643 tokens vs 16264), matching MLoC's 62.6% at full length.
• On ImageNet100 with 200 examples, EMLoC cuts inference FLOPs from 1.76T to 1.35T and inference time from 1866s to 1107s versus MLoC.
• EMLoC (54.9 average) edges out LoRA (54.7) and full fine-tuning (52.4) across ImageNet100, MME-RW and OK-VQA without updating weights.

Measured effects reported in the paper

Surprising findings

• Layer importance does not follow the pyramid pattern assumed by PyramidKV; certain early layers (e.g., layers 4, 8, 14 of Qwen2-VL) are critical and need more tokens.
• Empirically the global JS bound stays roughly constant across chunk counts K rather than growing linearly as the theory predicts.

Limitations

• Evaluated on a small set of benchmarks, with 100 randomly sampled test examples for datasets lacking validation splits.
• Theoretical bound relies on worst-case assumptions and an empirical independence assumption; the linear dependence on K is not observed empirically.
• Performance drops at extreme compression (15.7%, delta=0.02 gives 57.7% on ImageNet100).

Experimental setup

Datasets: ImageNet100 ScreenSpot MME-RW IllusionVQA OK-VQA YouCook2 VideoMME

Benchmarks: ImageNet100 ScreenSpot MME-RW IllusionVQA OK-VQA YouCook2 VideoMME

Metrics: accuracy context length inference FLOPs inference time peak memory

Baselines: MLoC RICES MTV LoRA Full fine-tuning M2PT VPT E2PT SnapKV H2O PyramidKV FastGen

Builds on: multi-modal long context learning (MLoC) PyramidKV SnapKV H2O FastV

Closest papers by meaning

• Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.86)
• CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning (AAAI 2026 · similarity 0.86)
• Online Adaptation of Language Models with a Memory of Amortized Contexts (NeurIPS 2024 · similarity 0.86)
• MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.86)
• MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.86)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.86)