AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models

Julianna Piskorz, Cristina Pinneri, Alvaro Correia, Motasem Alfarra, Risheek Garrepalli, Christos Louizos

ICML 2026 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
append many extra mask tokens to the promptLLaDA-Base and LLaDA-Instruct · few-shot ICL tasksaccuracy: 23 and 27 percentage points drop for base and instructHurt
append about 20 extra masksDream-Base and Dream-Instruct · few-shot ICL tasksaccuracy: 6 and 8 percentage points dropHurt
replace extra masks with repeated ' .' tokens (ablation)LLaDA-Base and LLaDA-Instruct · few-shot ICL tasksaccuracy: up to 3 and 10 percentage points drop vs 23 and 27 for masksHurt
add iterative unmasking (40 steps, high confidence) at inference (ablation)LLaDA with extra masks · few-shot ICL tasksaccuracyImproved
use high-confidence instead of random unmasking order (ablation)LLaDA with 40 decoding steps · few-shot ICL tasksaccuracyImproved
add CE+TV mask-agnostic fine-tuning (LoRA)LLaDA-Base-8B · few-shot ICL, 2/4/6 decoding stepsrelative degradation (%): 15.5% -> 7.9%Improved
add CE+TV mask-agnostic fine-tuning (LoRA)LLaDA-Instruct-8B · few-shot ICL, 2/4/6 decoding stepsrelative degradation (%): 27.5% -> 17.0%Improved
add the TV loss term to the CE loss (ablation)mask-agnostic fine-tuning of LLaDA · few-shot ICL tasksaccuracy robustness to extra masksImproved
add mask-agnostic fine-tuning (CE+TV)LLaDA · few-shot ICL taskslocality biasImproved
add MA-loss LoRA fine-tuningLLaDA-Base-8B · wikitext infillingROUGE-L: 0.497 -> 0.486No clear effect
add MA-loss LoRA fine-tuningLLaDA-Base-8B · lm1b infillingGenerative PPL: 6.058 -> 6.252No clear effect
add MA-loss LoRA fine-tuningLLaDA-MoE-Base · lm1b infillingGenerative PPL: 6.241 -> 5.828No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy