AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning

Jun Liu, Peilin Liu, Ruicheng Zhang, Zhang Senlei, Yanbo Chen, Ziao Wang, Jinyun Yang, mingqi wang, Shuhao Zhang, Xiaofei Liao, Hai Jin

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAGE, a dataflow-native framework for composing LLM inference pipelines (generation, retrieval, memory, context refinement, tool use) to enable modularity, controllability, and transparency under tail-latency and SLO constraints.

Problem

LLM applications run as multi-stage inference pipelines stitched together as RPC-connected services, which obscures cross-stage queueing and interference and limits pipeline-level compilation under tail-latency and SLO constraints.

Approach

Replace RPC-connected service stitching with a dataflow-native pipeline abstraction so that stages (generation, retrieval, memory, context refinement, tool use) are modular and the whole pipeline can be compiled and observed end to end. Details beyond the abstract are not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
swap FIFO for LoadAware-Spread schedulingSAGE scheduler · Mixed RAG, 16 nodesload balance: 52% -> 98%Improved
swap FIFO for LoadAware-Spread schedulingSAGE scheduler · Mixed RAG, 16 nodesP99 latency: 7.0 s -> 6.5 sImproved
swap FIFO for LoadAware-Spread schedulingSAGE scheduler · Mixed RAG, 16 nodesthroughput: 18.5/s -> 16.4/sHurt
swap FIFO for Priority schedulingSAGE scheduler · Mixed RAG, 16 nodesthroughput: 18.5/s -> 19.2/sImproved
swap FIFO for Priority schedulingSAGE scheduler · Mixed RAG, 16 nodesP99 latency: 7.0 s -> 12.0 sHurt
swap FIFO for RoundRobin schedulingSAGE scheduler · Mixed RAG, 16 nodesload balance: 52% -> 85%Improved
use LoadAware-Pack instead of FIFOSAGE scheduler · Mixed RAG, 16 nodesthroughput: 18.5/s -> 15.9/sHurt
add staggered admission controlmulti-pipeline contention · Multi-pipeline contentionP99 latency: 77 s -> 33 sImproved
add staggered admission controlmulti-pipeline contention · Multi-pipeline contentionaggregate throughput: 43.6 -> 30.7 tasks/sHurt
scale RAG pipeline from 1 to 16 nodesSAGE · RAGthroughput speedup: 11.4x at 16 nodesImproved
scale COMPUTE pipeline from 1 to 16 nodesSAGE · COMPUTEthroughput speedup: 3.8x at 16 nodesMixed
increase concurrency from 8 to 32SAGE COMPUTE pipeline · COMPUTE single nodethroughput: 22% of peak; P99 17xHurt

Later papers that compared with this method

2 later accepted papers report Sage as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Experimental setup

Metrics: tail latency SLO attainment

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy