Research map / Large language models / Reasoning and chain of thought

UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends the 4K context of LLaDA-8B to 128K via a diffusion-aware NTK RoPE rescaling plus 600 steps of post-training on PG19 packed long data with adaptive masking or EOD separators.

Problem

Diffusion LLMs have an unexplored long-context regime; naive RoPE extrapolation and training-free NTK scaling (LongLLaDA) fail beyond about 32K tokens, so the paper asks how to extend diffusion LLM context windows with post-training.

Approach

A diffusion-aware NTK scaling factor that sets the critical dimension using T_cap ≈ 2T_train and T_Ecap ≈ 2T_target to reflect bidirectional attention's symmetric relative-position coverage, combined with document-aware attention masking or EOD tokens during long-context post-training to limit cross-document interference.

Key results

• UltraLLaDA keeps perplexity at about 10.5-11.9 from 4K to 128K on PG19, whereas LLaDA-8B-Base rises to 343.88 at 128K.
• On RULER at 32K, UltraLLaDA averages 73.63 versus 5.69 for LongLLaDA; on LongBench-16K it averages 39.33 versus 35.38 for LongLLaDA.
• UltraLLaDA reports 100% NIAH retrieval accuracy at all lengths up to 128K, while LongLLaDA fails beyond 32K.

Surprising findings

• Direct concatenation scores slightly higher than boundary-aware strategies at 4K on RULER, though the advantage reverses at longer lengths.
• The training-free LongLLaDA baseline has higher perplexity than the base LLaDA at 4K and 16K.

Limitations

• The Diffusion-aware NTK formulation is motivated heuristically and the ablation gains are small (e.g., 39.13 vs 38.78 on LongBench).
• Comparisons use only LLaDA-8B and a single LongLLaDA baseline; no other context extension baselines are reported.
• Perplexity via Monte-Carlo denoising likelihood is not strictly comparable to autoregressive next-token perplexity.

Experimental setup

Datasets: PG19

Benchmarks: PG19 PPL-128K NIAH-128K LongBench RULER

Metrics: perplexity retrieval accuracy average score

Baselines: LLaDA-8B-Base LongLLaDA

Builds on: LLaDA NTK-aware scaling LongLLaDA RoPE FlashAttention2

Closest papers by meaning

• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.91)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.91)
• Large Language Diffusion Models (NeurIPS 2025 · similarity 0.91)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)