Research map / Large language models / Reasoning and chain of thought

DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DualMap, a dual-hash-based request scheduler for distributed LLM serving that maps each prompt prefix to two candidate instances and picks between them with SLO-aware routing, hotspot-aware migration, and dual-hash-ring scaling to jointly get cache affinity and load balance.

Problem

Cache-affinity scheduling (co-locating requests with shared prompt prefixes) conflicts with load balancing under skewed prefix popularity, hurting TTFT SLO attainment and effective request capacity in distributed LLM serving; existing schedulers operate in a single mapping space and only partially t

Approach

Map each request's prefix to two candidate instances via two independent hash functions (power-of-two-choices), choose between them by SLO-aware routing that favors cache reuse until TTFT threatens the SLO, migrate queued requests only within the candidate pair when overloaded, and scale with a dual-hash ring to limit remapping.

Key results

• DualMap improves effective request capacity by up to 2.25x under the same TTFT SLO compared with state-of-the-art scheduling (abstract).
• On Tool&Agent with Qwen2.5-7B, effective request capacity rises by up to 125% and goodput improves 16.7%-48% over the best baseline.
• Compared with the best baseline under high QPS, DualMap reduces P50 TTFT by 55.4%-97.4% and P90 TTFT by 82.3%-97%.

Surprising findings

• Cache Affinity alone performs poorly on Tool&Agent due to skewed prefixes, even at low QPS, showing that maximal cache reuse can violate SLOs.
• Increasing the number of choices beyond two yields diminishing load-balance returns while weakening cache locality, so d=2 is preferred.

Limitations

• Evaluated on trace-driven experiments with replayed Mooncake traces and a fixed 8-instance Ascend NPU cluster; scalability beyond that is simulated via Vidur.
• Relies on a TTFT estimator and a pending-prefill-token threshold to switch strategies; accuracy of the estimator is not deeply analyzed.
• Hash prefix length adaptivity requires tracking prefix hotness with sliding windows, adding design complexity.

Experimental setup

Datasets: Mooncake Conversation trace Mooncake Tool&Agent trace

Benchmarks: Conversation trace Tool&Agent trace

Metrics: effective request capacity goodput P50 TTFT P90 TTFT E2E latency cache hit rate load balance ratio (CV)

Baselines: Cache Affinity Least Loaded Min TTFT Preble

Builds on: power of two choices consistent hashing Cuckoo hashing vLLM Mooncake

Closest papers by meaning

• Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale (NeurIPS 2026 · similarity 0.89)
• MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.89)
• TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.87)
• OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.87)
• Preble: Efficient Distributed Prompt Scheduling for LLM Serving (ICLR 2025 · similarity 0.86)
• HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization (ICML 2026 · similarity 0.85)