Research map / Large language models / Reasoning and chain of thought

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents prima.cpp, a distributed on-device inference system that runs 30-70B Llama-family LLMs on heterogeneous consumer home clusters using pipelined-ring parallelism with prefetching and the Halda layer-assignment scheduler.

Problem

Running 30-70B LLMs on consumer home devices with insufficient RAM/VRAM, slow disks, Wi-Fi links and heterogeneous OSs, where existing distributed systems need enough aggregate memory or suffer from disk-offloading latency.

Approach

Pipelined-ring parallelism runs multiple rounds per token so devices prefetch different layer segments from disk via mmap, avoiding prefetch-release conflicts; Halda then solves a heterogeneity-aware ILP-based layer-to-device assignment (with iterative set assignment and device selection) for CPU/GPU workloads.

Key results

• On a four-device consumer cluster, prima.cpp reaches 674 ms/token TPOT for Llama 3-70B (Q4K), versus 10120 ms for llama.cpp and OOM for exo and dllama.
• Compared with llama.cpp, prima.cpp reduces TPOT by up to 17x and TTFT by up to 8x across 8-70B models.
• A 32B model with speculative decoding achieves 26 tokens/s; 70B with speculative decoding reaches 442 ms/token.

Surprising findings

• Vanilla prefetching fails for large models because fast disk loads evict earlier prefetched layers (prefetch-release conflict), so pipeline parallelism gives no benefit at k=1 for models above 60B.
• Heuristic layer partitioning by memory ratio (exo) can perform worse than schedules that account for compute and disk speed.

Limitations

• Targets single-request decoding; batching is only discussed in an appendix extension.
• Relies on Wi-Fi-connected consumer devices with 320-610 Mbps links; performance depends on disk speed and OS memory reclamation behavior.
• Analytical latency model uses per-device profiled coefficients and worst-case assumptions about prefetch timing.

Experimental setup

Metrics: TPOT (ms/token) TTFT (ms) memory pressure tokens/s

Baselines: llama.cpp exo dllama TPI-LLM

Builds on: llama.cpp pipeline parallelism speculative decoding HiGHS

Closest papers by meaning

• SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.88)
• HCInfer: An Efficient Inference System via Heterogeneous Error Compensation for Resource-Constrained Devices (NeurIPS 2026 · similarity 0.86)
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.86)
• A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.86)
• Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.86)
• ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive (NeurIPS 2025 · similarity 0.86)