Research map / Vision-language and multimodal / Multimodal LLMs

MoLoRA: Composable Specialization via Per-Token Adapter Routing

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes per-token routing of tokens to LoRA adapters (vocabulary-based or learned top-k gating, called MoLoRA), with a hot-set GPU memory design and CUDA graph kernels, claiming Qwen3-1.7B with four LoRAs beats Qwen3-8B on four reasoning benchmarks.

Problem

Multi-adapter serving systems route each whole sequence to a single LoRA adapter, which is inefficient for interleaved multimodal generation and suboptimal for mixed-capability requests spanning several domains.

Approach

Route individual tokens to adapters, either deterministically via vocabulary ranges (for unified multimodal tokenizers) or via a learned top-k router over LoRA experts (MoLoRA), reusing MoE-style dispatch and grouped GEMM infrastructure, with a static hot-set adapter memory layout enabling CUDA graph capture.

Key results

• Qwen3-1.7B with four LoRA adapters and learned routing exceeds Qwen3-8B on GSM8K (+14%), MATH (+8%), BBH (+2.5%), and GPQA (+2.1%) per the abstract/text.
• Per-token routing reduces 4 per-sequence passes to 1, giving a 4.1x latency speedup on a 4-modality kernel benchmark (2048 tokens, d=4096).
• CUDA graph capture with hot-set memory reduces P99 latency by 67x on average versus dynamic paging.

Surprising findings

• The learned router on the 1.7B model reportedly matches oracle domain routing, while a synthetic task shows MoLoRA recovers only 35% of the oracle improvement in validation loss.
• Scalar BGMV kernels from S-LoRA outperform the proposed tensor-core kernel below batch size ~160.

Limitations

• Reasoning results rely on a filtered general-purpose corpus whose details (size, domain labels, adapter training specifics) are not stated, and the comparison is only against Qwen3-8B.
• Top-k MoLoRA inference overhead grows linearly with k (up to 48.8x at k=4 in the synthetic setup), and end-to-end speedup is moderated by base model compute.
• Many claims rest on synthetic tasks and kernel microbenchmarks rather than real multimodal generation quality.

Experimental setup

Datasets: GSM8K MATH BBH GPQA

Benchmarks: GSM8K MATH BBH GPQA

Metrics: accuracy latency P99 latency throughput speedup

Baselines: Qwen3-8B S-LoRA Punica MoLE LoRA Switch

Builds on: LoRA S-LoRA Punica Switch Transformer GRPO

Closest papers by meaning

• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025 · similarity 0.86)
• R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing (NeurIPS 2025 · similarity 0.86)
• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.86)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.86)
• Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.86)
• Logits are All We Need to Adapt Closed Models (ICML 2025 · similarity 0.85)