Research map / Large language models / Reasoning and chain of thought

HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HeraSys, a multi-workflow LLM serving system that fuses redundant subgraphs across concurrent workflows via semantic signatures and uses load-aware joint scheduling (SRTF-like priority, resource skewing, adaptive batching and pipeline decomposition) to cut tail latency and raise throughput.

Problem

Serving concurrent multi-tenant LLM agentic/RAG workflows treats each workflow in isolation, missing cross-workflow redundancy and causing head-of-line blocking, starvation, and tail latency under high concurrency.

Approach

Combine pre-execution fine-grained graph fusion (hash-based semantic node signatures with DAG edge redirection and cache reuse) with a runtime joint scheduler that mixes dynamic remaining-time SRTF priority, node topology contribution, and minimum resource reservation for long-query heavy nodes, adapting batch size and split factor to load.

Key results

• HeraSys reduces P99 latency by up to 2.17x versus baselines (Mixed Workload, concurrency 300) and P95 by up to 1.94x.
• Throughput reaches up to 1.50x that of Ayo and nearly 2x that of LangGraph and LlamaIndex under latency constraints.
• Full HeraSys achieves 1.16x the total throughput of the variant without graph fusion at concurrency 100 on the mixed workload.

Surprising findings

• Pure SRTF reduces median latency but increases P99 latency by 22.3% versus full HeraSys, showing SRTF alone starves long tasks.
• Under high load, HeraSys reduces the average node count per query (19.0 to 15.6) to lower scheduling overhead, the opposite of its low-load behavior.

Limitations

• Scheduler is greedy and online; global optimality not guaranteed, and starvation freedom is only heuristic (no aging term).
• Evaluated on a single 4-GPU RTX 4090 node with one model (Llama-3-8B) and Google Search API workloads.
• Several method parameters (alpha, gamma, delta, rho) are hand-set, with parameter sensitivity shown only on one workload.

Experimental setup

Datasets: MS MARCO TruthfulQA

Benchmarks: RAG w/ Rerank RAG w/o Rerank Web Search Mixed Workload

Metrics: P95 latency P99 latency average latency QPS throughput

Baselines: LangGraph LlamaIndex Ayo

Builds on: vLLM Ayo PLAS SRTF

Closest papers by meaning

• OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration (ICML 2026 · similarity 0.87)
• KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows (NeurIPS 2025 · similarity 0.87)
• Preble: Efficient Distributed Prompt Scheduling for LLM Serving (ICLR 2025 · similarity 0.86)
• MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.86)
• HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location (NeurIPS 2025 · similarity 0.86)
• Flow: Modularized Agentic Workflow Automation (ICLR 2025 · similarity 0.86)