Research map / Large language models / Language, safety and interpretability

Blink of an eye: a simple theory for feature localization in generative models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Develops a general, assumption-light theory of "critical windows" (short time intervals where output features are fixed) for stochastic localization samplers covering both diffusion and autoregressive models, validated on LLM reasoning benchmarks.

Problem

Explaining why certain output features (e.g., object class in diffusion, correctness or jailbreak mode in LLM generations) are determined within narrow windows of the generation process, with prior theory limited to specific diffusion settings or distributional assumptions.

Approach

Frames diffusion and autoregressive samplers as stochastic localization samplers and uses a forward-reverse experiment with a total-variation bound to show critical windows arise when the sampler transitions from a larger sub-mixture to a smaller one; the bound makes no distributional assumptions and is dimension-free.

Key results

• Critical windows (>0.5 jump in answer-consistency probability) occur in 42.2%, 16.6%, and 30.2% of MATH generations for Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Phi-3-7B-Instruct respectively.
• Generations with critical windows are substantially less accurate; for Llama-3.1-8B-Instruct, the accuracy drop is up to 0.73 (ARC Easy) and 0.41 on MATH.
• The improved bound is dimension-independent, replacing the polynomial-in-dimension bound of prior diffusion theory.

Measured effects reported in the paper

Surprising findings

• Critical windows appear in a substantial fraction of generations on math and reasoning benchmarks, and incorrect responses contain 11%-33% more critical windows than correct ones for Llama-3.1-8B-Instruct.

Limitations

• Theory does not cover non-localization samplers such as GANs and consistency models.
• Assumes the learned sampler's distribution; approximation error to the true data distribution is not addressed.
• Critical window locations are only computed analytically for toy models in many examples.

Experimental setup

Datasets: ARC Challenge ARC Easy AQuA-RAT LogiQA MMLU TruthfulQA MATH

Benchmarks: ARC Challenge ARC Easy AQuA-RAT LogiQA MMLU TruthfulQA MATH

Metrics: probability of same answer under truncation and resampling accuracy with vs without critical windows critical window frequency

Baselines: Li and Chen 2024 (LC24) Qi et al. 2024 jailbreak critical tokens

Builds on: stochastic localization (Montanari 2023) Li and Chen 2024 critical windows theory all-or-nothing phenomenon

Closest papers by meaning

• Unveiling Concept Attribution in Diffusion Models (NeurIPS 2025 · similarity 0.89)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.89)
• Critical windows: non-asymptotic theory for feature emergence in diffusion models (ICML 2024 · similarity 0.88)
• Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.88)
• Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.87)
• Localizing Knowledge in Diffusion Transformers (NeurIPS 2025 · similarity 0.87)