Research map / Large language models / Reasoning and chain of thought
Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hogwild! Inference, which runs multiple instances of the same LLM in parallel over a shared, concurrently updated KV cache using RoPE-based cache rotation, letting reasoning models collaborate zero-shot.
Problem
Existing parallel LLM inference schemes (voting, SoT-style fixed decomposition, asynchronous threads) impose a fixed collaboration framework that suits only some tasks and cannot re-plan; the paper asks whether LLM workers can coordinate on the fly without a predefined framework while avoiding costl
Approach
Run several workers of the same LLM in one batch that share a single KV cache: each worker sees other workers' tokens as they are generated by rotating KV blocks to worker-specific positions, implemented by rotating only the current query per block (RoPE property), with a chat-like step-structured cache layout and s1-style prompts; no fine-tuning is used.
Key results
- On LIMO with QwQ-32B, Hogwild! Inference reaches higher accuracy than the sequential baseline at equal sequential-step budgets, with larger gains as workers increase from 2 to 4.
- On QwQ-32B-AWQ on an NVIDIA L40S, Hogwild! generates tokens about 2x faster with 2 workers and about 3.2-3.6x faster with 4 workers.
- GPT-4o collaborativeness scores are much higher for token-wise synchronized Hogwild! than for independent generation (no sync) or step-wise sync.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hogwild! Inference instead of sequential generation | QwQ-32B-AWQ · LIMO | accuracy at a fixed time budget | Helped |
| Use Hogwild! Inference with 2 workers instead of sequential inference | QwQ-32B-AWQ · token generation | tokens per second: nearly twice as fast | Helped |
| Use Hogwild! Inference with 4 workers instead of sequential inference | QwQ-32B-AWQ · token generation | tokens per second: 3.2{-}3.6× faster | Helped |
| Use Hogwild! Inference instead of the single-worker baseline | QwQ-32B · LIMO | accuracy at a fixed number of consecutive steps | Helped |
| Use Hogwild! Inference instead of the single-worker baseline | QwQ-32B · GSM8K×5 | per-question accuracy | No clear effect |
| Use the combined cache layout rather than the contiguous layout (ablation) | Hogwild! Inference · LIMO | accuracy at longer reasoning budgets | Helped |
| Use token-wise synchronization rather than step-wise synchronization (ablation) | Hogwild! Inference · LIMO | accuracy at smaller budgets | Helped |
| Add the collaboration prompt asking whether the worker is doing redundant work (ablation) | Hogwild! Inference · LIMO | accuracy-budget trade-off | Helped |
| Use cache rotation instead of re-encoding worker tokens at new positions (ablation) | Hogwild! Inference · LIMO | accuracy | Mixed |
| Use multiple workers rather than a single worker (ablation) | Hogwild! Inference · LIMO, smaller budgets | accuracy | Helped |
| Synchronize workers token-wise rather than only after completed reasoning steps (ablation) | Hogwild! Inference · LIMO | GPT-4o collaborativeness score | Helped |
Surprising findings
- Off-the-shelf reasoning models (QwQ-32B, Qwen3, Phi-4-R+, DeepSeek-R1) collaborate via the shared cache without any fine-tuning.
- Qwen3-1.7B and to a lesser extent Qwen3-4B get distracted and fail to adapt to the shared-cache collaboration setup.
Limitations
- Reduced robustness on smaller models and longer contexts.
- Collaboration quality is judged by a proprietary model (GPT-4o), limiting reproducibility.
- Relies on prompting; models often but not always detect redundant work.
Experimental setup
Datasets: GSM8K LIMO LiveCodeBench OlympiadBench AIME'25
Benchmarks: GSM8K x5 (synthetic) LIMO LiveCodeBench v5 OlympiadBench AIME'25
Metrics: accuracy vs sequential token budget Pass@1 collaborativeness score (1-6) tokens per second latency
Baselines: Baseline (sequential with early stopping) Skeleton-of-Thought Self-consistency
Builds on: Self-Consistency Skeleton-of-Thought Hogwild! SGD Flash-Decoding PagedAttention s1
Closest papers by meaning
- RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse (ICML 2026 · similarity 0.89)
- Bag of Tricks for Inference-time Computation of LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries (NeurIPS 2025 · similarity 0.89)
- Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation (ICLR 2024 · similarity 0.89)
- Cache-to-Cache: Direct Semantic Communication Between Large Language Models (ICLR 2026 · similarity 0.89)
- A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.