Research map / Large language models / Reasoning and chain of thought
Cost-efficient Collaboration between On-device and Cloud Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MINION and MINIONS, protocols where a small on-device LM with local data collaborates with a cloud frontier LM, cutting remote cost ~5.7-30x while recovering most of frontier-model accuracy on long-document reasoning.
Problem
Cloud frontier LMs are expensive for data-intensive reasoning over long documents, while small on-device LMs are underused for such tasks. The paper asks whether local-remote collaboration can cut cloud inference cost while preserving quality.
Approach
A naive free-form chat (MINION) where only the local model reads the full context, and an extension (MINIONS) where the cloud model writes Python code that decomposes the task into single-step jobs over context chunks, executed in parallel on the local model, with filtered results aggregated by the cloud model over multiple rounds.
Key results
- MINION reduces remote cost by 30.4x and recovers 87% of GPT-4o performance (macro average).
- MINIONS with an 8B local model recovers 97.9% of remote-only performance at 18.0% of cloud cost (about 5.7x cheaper on average).
- With a 3B local model, MINIONS achieves 93.4% of remote-only performance at 16.6% of cloud cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| naive chat protocol (Minion) between Llama-8B local and GPT-4o remote | local-remote collaboration · Macro avg FinanceBench/LongHealth/QASPER | remote cost reduction: 30.4x reduction, recovers 87% of remote performance | Mixed |
| decomposition into single-step jobs over context chunks executed in parallel locally (MinionS) vs naive Minion | Llama-8B local with GPT-4o remote · Macro avg FinanceBench/LongHealth/QASPER | accuracy: 0.630 -> 0.709 | Helped |
| decomposition into parallel local jobs (MinionS) with 8B local model | local-remote system · Macro avg | cost (USD/query) relative to remote-only: $0.233 remote-only vs $0.042 MinionS | Helped |
| increasing number of tasks per round from 1 to 16 (ablation) | MinionS · LongHealth and QASPER | accuracy: up to 14 points | Helped |
| increasing samples per task from 1 to 32 (ablation) | MinionS · LongHealth and QASPER | accuracy: +7.4 points on average, but worse beyond 16 samples; 5x remote prefill cost | Mixed |
| decreasing chunk size from 100 pages to 5 pages per chunk (ablation) | MinionS · LongHealth and QASPER | accuracy: +11.7 points, 2.41x remote prefill cost | Helped |
| increasing max communication rounds from 1 to 5 in Minion (ablation) | Minion with Llama-3.2 local · Macro avg | accuracy: +8.5 points average | Helped |
| scratchpad strategy for maintaining context across MinionS rounds vs simple retries (ablation) | MinionS · Macro avg | accuracy vs cost tradeoff: scratchpad slightly better cost-accuracy tradeoff | Mixed |
| local model size 1B to 3B to 8B in MinionS (ablation) | MinionS · Macro avg | fraction of GPT-4o-only performance recovered: 49.5% (1B), 93.4% (3B), 97.9% (8B) | Helped |
Surprising findings
- Splitting multi-part instructions into separate requests improved small-model performance by 56 points.
- Increasing context length from under 1K to over 65K tokens reduced a 3B local model's extraction performance by 13%.
Limitations
- Local model costs treated as free; hardware and energy costs ignored, and latency not empirically studied.
- Evaluated on modified long-context datasets with added distractor documents, and only on a subset for ablations (128 problems).
- RAG comparison and summarization evaluation are limited; privacy not addressed.
Experimental setup
Datasets: FinanceBench LongHealth QASPER BooookScore
Benchmarks: FinanceBench LongHealth QASPER BooookScore
Metrics: accuracy remote API cost in USD cost reduction factor
Baselines: GPT-4o remote-only Llama-3.2 3B local-only Llama-3.1 8B local-only MINION RAG with BM25 RAG with OpenAI text-embedding-3-small
Builds on: chain-of-thought retrieval-augmented generation MemGPT information bottleneck
Closest papers by meaning
- Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training (ICML 2026 · similarity 0.88)
- Intelligence per Watt: Measuring Intelligence Efficiency of Local AI (NeurIPS 2026 · similarity 0.88)
- HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference (ICML 2026 · similarity 0.85)
- Collaborative LLM Numerical Reasoning with Local Data Protection (AAAI 2026 · similarity 0.84)
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024 · similarity 0.84)
- Cloud-Device Collaborative Learning for Multimodal Large Language Models (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.