Research map / Large language models / Reasoning and chain of thought
SAGE: A Dataflow-Native Framework for Modular, Controllable, and Transparent LLM-Augmented Reasoning
Jun Liu, Peilin Liu, Ruicheng Zhang, Zhang Senlei, Yanbo Chen, Ziao Wang, Jinyun Yang, mingqi wang, Shuhao Zhang, Xiaofei Liao, Hai Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAGE, a dataflow-native framework for composing LLM inference pipelines (generation, retrieval, memory, context refinement, tool use) to enable modularity, controllability, and transparency under tail-latency and SLO constraints.
Problem
LLM applications run as multi-stage inference pipelines stitched together as RPC-connected services, which obscures cross-stage queueing and interference and limits pipeline-level compilation under tail-latency and SLO constraints.
Approach
Replace RPC-connected service stitching with a dataflow-native pipeline abstraction so that stages (generation, retrieval, memory, context refinement, tool use) are modular and the whole pipeline can be compiled and observed end to end. Details beyond the abstract are not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| swap FIFO for LoadAware-Spread scheduling | SAGE scheduler · Mixed RAG, 16 nodes | load balance: 52% -> 98% | Improved |
| swap FIFO for LoadAware-Spread scheduling | SAGE scheduler · Mixed RAG, 16 nodes | P99 latency: 7.0 s -> 6.5 s | Improved |
| swap FIFO for LoadAware-Spread scheduling | SAGE scheduler · Mixed RAG, 16 nodes | throughput: 18.5/s -> 16.4/s | Hurt |
| swap FIFO for Priority scheduling | SAGE scheduler · Mixed RAG, 16 nodes | throughput: 18.5/s -> 19.2/s | Improved |
| swap FIFO for Priority scheduling | SAGE scheduler · Mixed RAG, 16 nodes | P99 latency: 7.0 s -> 12.0 s | Hurt |
| swap FIFO for RoundRobin scheduling | SAGE scheduler · Mixed RAG, 16 nodes | load balance: 52% -> 85% | Improved |
| use LoadAware-Pack instead of FIFO | SAGE scheduler · Mixed RAG, 16 nodes | throughput: 18.5/s -> 15.9/s | Hurt |
| add staggered admission control | multi-pipeline contention · Multi-pipeline contention | P99 latency: 77 s -> 33 s | Improved |
| add staggered admission control | multi-pipeline contention · Multi-pipeline contention | aggregate throughput: 43.6 -> 30.7 tasks/s | Hurt |
| scale RAG pipeline from 1 to 16 nodes | SAGE · RAG | throughput speedup: 11.4x at 16 nodes | Improved |
| scale COMPUTE pipeline from 1 to 16 nodes | SAGE · COMPUTE | throughput speedup: 3.8x at 16 nodes | Mixed |
| increase concurrency from 8 to 32 | SAGE COMPUTE pipeline · COMPUTE single node | throughput: 22% of peak; P99 17x | Hurt |
Later papers that compared with this method
2 later accepted papers report Sage as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- LLMs Unleashed: Generating Protocol Code from RFC Specifications (AAAI 2026 · better on 3 of 3 shared results)
- Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning (NeurIPS 2026 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Metrics: tail latency SLO attainment
Closest papers by meaning
- ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning (ICML 2026 · similarity 0.89)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.87)
- Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.87)
- AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving (ICML 2026 · similarity 0.87)
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (NeurIPS 2025 · similarity 0.87)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
