Research map / Architectures and efficiency / RNNs and normalization

Efficiently Serving Large Multimodal Models Using EPD Disaggregation

ICML 2025 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EPD disaggregation, which separates multimodal encoding, prefill, and decode onto dedicated GPU resources with multimodal-token caching, intra-request parallelism, a configuration optimizer, and dynamic role switching to improve LMM serving SLOs and memory efficiency.

Problem

Serving large multimodal models is hard to meet TTFT and TPOT SLOs because the added multimodal encoding stage adds compute and memory overhead and inflates tokens, and aggregated systems co-locate encoding with prefill and decode.

Approach

Decouple the encode (vision/audio encoder), prefill (LLM), and decode stages onto independent instances, transfer multimodal tokens asynchronously via an MM cache, parallelize a single request's image patches across encoding workers (IRP), tune configs with Bayesian optimization on a simulator, and switch instance roles dynamically under changing workloads.

Key results

• EPD reduces TTFT by up to 71.9% vs DistServe for MiniCPM-V 2.6 and supports up to 10x more images per request for InternVL2-8B at 4032x3024 resolution.
• EPD supports a 22x larger encoding batch size than DistServe for InternVL2-26B at 787x444 resolution (22 vs 1).
• On the NextQA workload EPD is the only system reaching 90% SLO attainment at low request rates; ablation shows dynamic role switching cuts latency by 2.2x versus no switching.

Measured effects reported in the paper

Surprising findings

• Disabling IRP increases TTFT by up to 2.9x at 6 images per request, showing intra-request parallelism is a major contributor.
• Limitations section notes that aggregated serving can outperform disaggregated serving when throughput per dollar is the goal and SLOs are relaxed.

Limitations

• Disaggregation can underperform aggregated serving in throughput-per-dollar settings with relaxed SLOs due to inter-stage communication and pipeline bubbles.
• Gains depend on careful tuning of resource allocation and worker migration to balance stages.
• Visual token migration can be a bottleneck, which the paper leaves to future compression work.

Experimental setup

Datasets: NextQA Video-MME synthetic workload

Benchmarks: NextQA Video-MME synthetic workload

Metrics: TTFT TPOT SLO attainment goodput maximum batch size images per request KV cache size throughput

Baselines: DistServe vLLM

Builds on: DistServe vLLM DéjàVu

Closest papers by meaning

• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism (NeurIPS 2025 · similarity 0.89)
• DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.88)
• Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.87)
• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.87)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.87)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)