Research map / Large language models / Language, safety and interpretability

Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Multiverse, a MapReduce-style natively parallel generative LLM built via data curation (Multiverse Curator/Multiverse-1K), attention modification (Multiverse Attention), and an inference engine (Multiverse Engine), fine-tuning Qwen2.5-32B-Instruct into Multiverse-32B.

Problem

Autoregressive LLMs generate sequentially, limiting latency; existing parallel approaches (diffusion models, external tool-based width scaling) either ignore logical dependencies or lose internal state, so the paper seeks adaptive, lossless internal parallel generation.

Approach

Internalize a MapReduce paradigm in the generated sequence (Map task decomposition, parallel Process paths, Reduce merge) via control tags, using a Multiverse Attention mask with shared start positions per parallel block so branches are trained in parallel and KV states are losslessly merged in a specialized serving engine built on SGLang.

Key results

• Multiverse-32B reaches 53.8% on AIME24 and 45.8% on AIME25, comparable to Autoregressive-32B (54.6% and 45.0%) and well above Qwen2.5-32B-Instruct (15.8% and 10.4%).
• Multiverse-32B achieves up to 2x wall-clock speedup per generated token, with speedup scaling roughly linearly with degree of parallelism across batch sizes 1 to 128.
• Under fixed 1K-4K budgets, Multiverse-32B gains 2.23% on GPQA-Diamond and 1.51% on MATH500 over the AR baseline.

Measured effects reported in the paper

Surprising findings

• Over 98% of s1K-1.1 CoT trajectories contain parallelizable branches, yet a probing classifier on AR-LLM hidden states performs near random at identifying where parallel blocks start.
• Prompting reveals about a 90% gap between implicit parallel structures in CoT and those explicitly generated when asked.

Limitations

• Application is limited to LLM reasoning tasks; generalization to other data and tasks is underexplored.
• Trained only with SFT; parallelism is limited on long generations (e.g., AIME), partly due to scarce >16K token training data.
• Requires a specialized Multiverse engine for parallel execution.

Experimental setup

Datasets: s1K-1.1 Multiverse-1K

Benchmarks: AIME24 AIME25 MATH500 GPQA-Diamond

Metrics: pass@1 number of parallel (generated tokens / sequential tokens) latency per token speedup

Baselines: Qwen2.5-32B-Instruct Autoregressive-32B s1-32B s1.1-32B

Builds on: Qwen2.5-32B-Instruct SGLang APE s1

Closest papers by meaning

• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models (ICML 2026 · similarity 0.88)
• Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (NeurIPS 2025 · similarity 0.87)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.85)
• ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.85)
• Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.85)
• Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows (NeurIPS 2026 · similarity 0.85)