Research map / Large language models / Reasoning and chain of thought

Generalized Parallel Scaling with Interdependent Generations

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Bridge, a small batch-axis attention block added to LLMs so that parallel responses to the same prompt share information mid-generation, improving RLVR-trained reasoning accuracy and response-set coverage.

Problem

Parallel LLM inference scaling samples N responses for one prompt independently, so the responses cannot exploit information from each other, limiting accuracy and set-level quality.

Approach

Treat hidden states as a B x S x D tensor and add a Bridge block that applies attention across the batch axis (samples sharing a prompt) at each token timestep, without positional encoding or KV cache, adding only 2.8-5.1% parameters while keeping next-token sampling parallel.

Key results

• On DS-Qwen-7B, Bridge averages 39.40 across 7 math benchmarks vs 37.75 for RLVR only (+5.85 over base vs +4.20), a relative 39% larger gain than the next best method.
• On DS-Qwen-7B, all-correct rate on a single competition problem rises from 15.0% to 17.8%.
• Bridge adds 2.8% to 5.1% parameters and, trained at width 4, still outperforms independent sampling at width 8 on accuracy and G-Pass@8 metrics.

Surprising findings

• Bridge blocks contribute little to residual feature norms compared to P-Match blocks despite large performance gains.
• Cold-start RLVR without SFT warm-up already performs nearly as well as warmed-up Bridge.

Limitations

• Gains reported mainly on distilled DeepSeek-R1 models of 1.5B-8B size; scaling to larger models untested.
• Training only on math; non-math results are mixed (e.g., DS-Llama-8B XSum and CNN/DailyMail drop).
• Single-group GRPO setup; unbiasedness and independence considerations for multiple groups left to future work.

Experimental setup

Datasets: GSM8K DeepScaleR-Preview-Dataset

Benchmarks: MATH-500 AIME24 AIME25 AMC23 BRUMO25 CMIMC25 HMMT_FEB25 XSum CNN/DailyMail GPQA ZebraLogic Countdown

Metrics: accuracy (Pass@1) G-Pass@k_tau coverage Rouge-1 BERTScore variance self-consistency accuracy

Baselines: RLVR only P-Match Original DS-Qwen/DS-Llama models ParScale

Builds on: GRPO axial attention Pass@k training G-Pass@k ParScale Hogwild! Inference

Closest papers by meaning

• Predicting LLM Output Length via Entropy-Guided Representations (ICLR 2026 · similarity 0.88)
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.88)
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.87)
• Compute Aligned Training: Optimizing for Test Time Inference (NeurIPS 2026 · similarity 0.87)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.87)
• Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models (ICML 2026 · similarity 0.87)