Research map / Large language models / Language, safety and interpretability

FLARE: Diffusion for Hybrid Language Model

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents FLARE, a recipe converting hybrid softmax+linear-attention Qwen3.5 AR checkpoints into diffusion LLMs (2B/4B/9B) via a clean/noisy two-stream loss, with hardware-aware recurrent-state kernels and a unified AR-Trust/Diffusion-Trust serving stack.

Problem

AR-to-diffusion LLM conversion often loses the seed model's capability, and it is unclear how to make diffusion training and serving efficient on hybrid-attention (softmax plus linear-attention) backbones under a limited training budget.

Approach

Jointly train a causal clean stream (next-token loss) and a block-bidirectional noisy stream (two complementary masked views, token-balanced loss) with a document-packed mask; realize noisy-block visibility on GDN layers via recurrent-state handoff from clean block-boundary states rather than dense masking, and serve both verified speculative (AR-Trust) and parallel denoising (Diffusion-Trust) fro

Key results

• FLARE-9B (AR-Trust) reaches 95.20 on MATH-500 vs 96.60 for Qwen3.5-9B and 49.71 on LiveCodeBench v6, exceeding LLaDA-2.1-flash on GPQA-Diamond (71.21 vs 66.67).
• On a single A100-80GB at concurrency 8, FLARE-2B reaches 2,087 tokens/s on GSM8K, a 2.2x gain over LLaDA-2.1-mini.
• Training MFU of FLARE-2B at B=4 on 8xA100 rises to 24.81%, slightly above the AR Qwen3.5-2B reference (24.04%).

Surprising findings

• Diffusion-Trust decoding is consistently weaker on code generation than AR-Trust, which the authors attribute partly to answer-extraction and truncation issues.
• Automatic IFD-based data filtering does not close the residual gap to the source AR model.

Limitations

• Training roughly doubles per-step compute and memory because clean and noisy views are concatenated into a 2L input.
• Residual gap to the source Qwen3.5 AR checkpoints on instruction following and some coding benchmarks.
• Validated only on dense models under 10B parameters with a single SFT stage; MoE and RL post-training untested.

Experimental setup

Datasets: Llama-Nemotron-Post-Training Nemotron-Math-Proofs-v1 Nemotron-Cascade-1/-2-SFT-Data Nemotron-Instruction-Following-Chat-v1/v2 Nemotron-Post-Training-Dataset-v2 GSM8K MATH-500 AIME 24/25 ARC-Challenge GPQA-Diamond MMLU MMLU-Pro

Benchmarks: GSM8K MATH-500 AIME 24 AIME 25 ARC-Challenge GPQA-Diamond MMLU MMLU-Pro IFEval HumanEval MBPP LiveCodeBench v6

Metrics: accuracy pass@1 tokens per second MFU

Baselines: Mercury-2 LLaDA-2.0-mini LLaDA-2.1-mini LLaDA-2.0-flash LLaDA-2.1-flash SDAR-30B-A3B SDAR-1.7B SDAR-4B SDAR-8B Qwen3.5

Builds on: Gated DeltaNet Qwen3.5 BD3-LM SDAR TiDAR I-DLM

Closest papers by meaning

• SimSD: Simple Speculative Decoding in Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.89)
• Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs (NeurIPS 2026 · similarity 0.88)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.88)
• When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models (ICML 2024 · similarity 0.88)
• DAWN: Dependency-Aware Fast Inference for Diffusion LLMs (NeurIPS 2026 · similarity 0.88)