Research map / Large language models / Reasoning and chain of thought
SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpecEdge, an edge-assisted LLM serving framework that offloads speculative drafting to consumer GPUs at the edge while the server only verifies, using proactive edge drafting and pipeline-aware server scheduling.
Problem
Serving LLMs at scale is costly, and existing server-centric systems (and layer-split approaches) fail to exploit cheap consumer-grade edge GPUs over wide-area networks due to latency and I/O constraints.
Approach
Disaggregate speculative decoding so edge GPUs draft tokens and the server only verifies, exchanging only token outputs; edge keeps drafting proactively during verification (complete draft alignment reuse), and the server interleaves verification requests from multiple users to keep GPUs busy.
Key results
- SpecEdge achieves 1.91x average cost efficiency and 2.22x average server throughput over server-only speculative decoding on SpecBench.
- Inter token latency is reduced by 11.24% on average versus server-only baseline, even at 14.07 ms RTT.
- Layer-split decoding is 2.73x slower than SpecEdge at 15 ms RTT (99.63 ms vs 36.47 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Offload drafting to edge GPUs and use proactive drafting with pipeline-aware scheduling | Server-only speculative decoding · SpecBench | cost efficiency: 1.91× | Helped |
| Offload drafting to edge GPUs and use proactive drafting with pipeline-aware scheduling | Server-only speculative decoding · SpecBench | server throughput: 2.22× | Helped |
| Offload drafting to edge GPUs and use proactive drafting with pipeline-aware scheduling | Server-only speculative decoding · SpecBench | inter-token latency: 11.24% | Helped |
| Add proactive edge drafting to basic disaggregation (ablation) | SpecEdge disaggregated drafting and verification · SpecBench, Qwen3-14B/1.7B | inter-token latency: 28 ms | Helped |
| Add pipeline-aware scheduling to disaggregation with proactive edge drafting (ablation) | SpecEdge · SpecBench, Qwen3-14B/1.7B | server throughput: 32.76 tokens/s -> ~67.89 tok/s; 2.07× improvement | Helped |
| Add proactive edge drafting (ablation) | Server-only speculative decoding · SpecBench, three target–draft pairs | generated tokens per verification: 13.21% higher | Helped |
| Batch concurrent requests on one edge GPU instead of dedicating an edge GPU per request (ablation) | SpecEdge · Qwen3-14B/1.7B, batch size 2–4 | cost efficiency: 4.4% to 29.5% better | Helped |
| Batch concurrent requests on one edge GPU instead of dedicating an edge GPU per request (ablation) | SpecEdge · Qwen3-14B/1.7B, batch size 2–4 | inter-token latency: 5.9% to 19.0% slower | Hurt |
| Enable reasoning mode (ablation) | SpecEdge · Qwen3-14B/1.7B, Qwen3-14B/0.6B, and Qwen3-32B/1.7B | server throughput: 64.343 -> 72.249; 70.703 -> 81.890; 24.880 -> 27.617 | Helped |
| Replace tree-based drafting with a linear candidate sequence (ablation) | SpecEdge · Llama2-13B/TinyLlama-1.1B | server throughput: 1.96× higher | Helped |
| Replace tree-based drafting with a linear candidate sequence (ablation) | SpecEdge · Llama2-13B/JackFram-160M | inter-token latency: 30.530 -> 30.818 | Hurt |
| Use a lighter RTX 3060 Ti edge GPU instead of the RTX 4090 (ablation) | SpecEdge · Qwen3-14B/1.7B | server throughput: 30.816 -> 50.297 | Helped |
Surprising findings
- Naively expanding all leaf nodes for proactive drafting yields negligible gains; expanding only the single highest-log-probability path works better.
Limitations
- Cost analysis assumes rented consumer GPUs from edge cloud providers; untrusted user-owned devices raise fault-tolerance and security issues not addressed.
- Evaluated only on models up to 33B; larger models left to future work.
- Batch size 1 is the main setting; batched drafting trades latency for cost.
Experimental setup
Datasets: SpecBench C4 OpenAssistant (OAsst) WikiText-2 MTBench
Benchmarks: SpecBench C4 OAsst WikiText-2 MTBench
Metrics: server throughput (tok/s) cost efficiency (1k tokens/$) inter token latency (ms) generated tokens per verification speedup
Baselines: server-only tree-based speculative decoding autoregressive decoding layer-split decoding Petals
Builds on: speculative decoding tree-based speculative decoding Sequoia SpecExec Petals
Closest papers by meaning
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.88)
- TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.88)
- HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference (ICML 2026 · similarity 0.87)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.87)
- MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.87)
- A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.