Research map / Vision-language and multimodal / Multimodal LLMs

ROSE: Rotate Your Large Language Model to See

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ROSE, a 7B MLLM that injects visual information by rotating pretrained LLM weight matrices with image-conditioned orthogonal (Cayley-transformed) matrices instead of adding visual tokens to the input sequence.

Problem

Input-space visual token injection in MLLMs greatly increases context length and quadratic compute, and fine-tuning on vision-dominant sequences can erode pretrained linguistic capabilities.

Approach

Shift visual injection from the input space to the parameter space: a vision-guided rotation matrix generator (VRMG) produces block-diagonal orthogonal matrices, via Cayley transform, that left-multiply each linear projection's weights; since rotations preserve pairwise angles between weight vectors, pretrained geometry is kept while visual semantics modulate the LLM.

Key results

• ROSE-7B averages 74.5 across 12 multimodal benchmarks vs 76.0 for Qwen2.5-VL-7B, with 80.7% lower FLOPs (3.7 vs 19.2 T) and 56.4% lower latency per the abstract.
• Compared with input-space LLaVA-style injection under identical data, ROSE-style achieves 66.7 average vs 65.3 on language and multimodal benchmarks, using 13.8% of inference FLOPs and 30% of training time.
• Across vision-token lengths 512 to 32768, ROSE reduces FLOPs by an average of 79.9% and latency by 58.7% vs Qwen2.5-VL-7B.

Surprising findings

• Preserving only the direction (not magnitude) of weight vectors retains much more MMLU accuracy than preserving magnitude, motivating direction-based injection.

Limitations

• Average benchmark score (74.5) is below Qwen2.5-VL-7B (76.0), so the gain is in efficiency rather than accuracy.
• Adds about 200M parameters via VRMG modules and relies on a fixed block-diagonal sparsification of the rotation matrices.

Experimental setup

Datasets: LAION-2B DataComp-1B FineVision LLaVA-665K

Benchmarks: MMBench MMMU SEED-Image MME MMStar TextVQA ChartQA DocVQA InfoVQA AI2D RealWorldQA GQA

Metrics: accuracy FLOPs inference latency GPU memory

Baselines: LLaVA-OV-0.5B Qwen2-VL-2B Qwen2.5-VL-3B Gemma3-4B Phi-3.5-V-4B LLaVA-NeXT-7B LLaVA-OV-7B Qwen2-VL-7B Qwen2.5-VL-7B Gemma3-12B CogVLM2-19B LLaVA-style input-space injection

Builds on: LLaVA Qwen2.5-7B SigLIP2 LMMs-eval

Closest papers by meaning

• Towards Multimodal Large Language Models with Both Training and Inference Efficiency (ICML 2026 · similarity 0.90)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.90)
• Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.90)
• HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.90)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
• IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (AAAI 2025 · similarity 0.89)