Research map / Vision-language and multimodal / Multimodal LLMs

Towards Multimodal Large Language Models with Both Training and Inference Efficiency

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAISA, an MLLM architecture whose self-attention blocks skip attention among visual tokens (NAAViT) and whose FFNs skip visual tokens, with per-layer MLP projectors aligning visual features to each layer's self-attention input space, cutting LLaVA-1.5-level inference FLOPs by 66% and training budget by 26%.

Problem

MLLMs face a trade-off: embedding-space alignment (LLaVA) is training-efficient but inference-inefficient due to long visual token sequences, while cross-attention alignment (Flamingo) is inference-efficient but needs many new parameters and lots of data to train.

Approach

Eliminate attention among visual tokens (NAAViT: only text tokens act as queries, visual tokens are not updated) and skip FFNs on visual tokens, then align visual features directly to the input spaces of each LLM layer's self-attention via a per-layer lightweight projector, so the LLM is reused without adding cross-attention blocks.

Key results

• SAISA with Vicuna-7B and CLIP-ViT-L/14-336 cuts inference TFLOPs from 8.53 to 2.86 (66% reduction) while reaching 62.4 average vs 61.0 for LLaVA-1.5 across the ablation benchmarks.
• SAISA reports a training budget of about 80 GPU hours, a 26% reduction relative to LLaVA-1.5 as stated in the introduction.
• SAISA beats LLaVA-1.5 on 3 of 5 comprehensive benchmarks (MMMU 36.9 vs 35.7, MMBench EN 65.7 vs 64.3, CN 59.0 vs 58.3) and on MMVP (26.0 vs 24.7).

Surprising findings

• Eliminating attention among visual tokens (NAAViT) outperforms vanilla self-attention in the pilot experiment on LLaVA-1.5 (e.g., OK-VQA 56.0 vs 53.4).
• SAISA with linear projector layers achieves performance comparable to LLaVA-1.5 with MLP projection.

Limitations

• Per-layer MLP projector adds non-negligible parameters that the authors acknowledge as a limitation.
• Not evaluated on high-resolution images, multiple images, interleaved content, or video.
• Limited capability on complex real-world instructions and domain-specific tasks such as medicine.

Experimental setup

Datasets: LLaVA pre-train 558k LLaVA-1.5 mixture 665k MMMU MME MMBench MMBench-CN SEED-Bench POPE GQA ScienceQA IMG TextVQA OK-VQA

Benchmarks: MMMU MME MMBench MMBench-CN SEED-Bench POPE GQA ScienceQA IMG TextVQA OK-VQA MMVP CV-Bench

Metrics: accuracy inference TFLOPs training data volume inference prefill latency (ms)

Baselines: LLaVA-1.5 Flamingo OpenFlamingo Qwen-VL-Chat IDEFICS InstructBLIP BLIP-2 MiniGPT-4 Otter Shikra MiniGPT-v2

Builds on: LLaVA-1.5 Flamingo Vicuna CLIP GQA

Closest papers by meaning

• ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.91)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
• ROSE: Rotate Your Large Language Model to See (CVPR 2026 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.90)