Research map / Large language models / Language, safety and interpretability

MARS: Enabling Autoregressive Models Multi-Token Generation

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Fine-tunes an instruction-tuned AR model (Qwen2.5 0.5B/7B) with a clean-plus-masked-block stream and an auxiliary AR loss so it can optionally emit multiple tokens per forward pass while keeping one-token quality.

Problem

Autoregressive LLMs generate one token per forward pass even when consecutive tokens are predictable; existing multi-token methods either add extra heads/draft models or convert the model into block diffusion models that degrade reasoning and coding quality.

Approach

Keep the model strictly causal, right-shifted and left-to-right, and fine-tune it on a concatenated clean and [MASK]-block stream with a structured attention mask plus an auxiliary AR loss on the clean stream, so the AR signal does not decay with block size; inference uses confidence-thresholded left-to-right acceptance over a sliding window of masks.

Key results

• At one token per step, MARS-7B averages 58.1 vs 56.6 for AR SFT across six benchmarks; MARS-0.5B averages 30.4 vs 28.7.
• At tau=0.95 on Qwen2.5-7B, MARS (B=4) averages 56.8 with 1.68 tokens per forward, still above AR SFT.
• With block-level KV cache on GSM8K at batch size 4, MARS reaches 1.71x wall-clock speedup over AR with KV cache on Qwen2.5-7B.

Surprising findings

• Without the auxiliary SFT loss, larger block sizes sharply degrade quality (avg 28.4 to 22.2 from B=4 to B=16), whereas with it performance is stable.
• Block Diffusion with bidirectional intra-block attention collapses on BBH (7.5) and MMLU-Pro (2.0) at 0.5B.

Limitations

• Aggressive thresholds (tau < 0.7) cause substantial quality loss.
• Block-level KV cache requires batch synchronization at block boundaries, limiting gains at large batch sizes.
• 7B model evaluated only at B=4 due to compute constraints; training doubles sequence length and cost.

Experimental setup

Datasets: Dolci-Instruct-SFT

Benchmarks: IFEval BBH MMLU-Pro GPQA GSM8K HumanEval

Metrics: accuracy pass@1 tokens per forward wall-clock throughput speedup

Baselines: AR SFT Block Diffusion Jacobi decoding Medusa EAGLE

Builds on: Block Diffusion Qwen2.5-Instruct Multi-token prediction speculative decoding

Closest papers by meaning

• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion (NeurIPS 2026 · similarity 0.90)
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.90)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.88)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.88)
• Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs (NeurIPS 2026 · similarity 0.88)
• Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.88)