Research map / Image generation / Fast diffusion sampling

Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Training-free diffusion acceleration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LoTTS, a training-free localized test-time scaling method for diffusion and flow text-to-image models that uses prompt-contrastive cross-attention to build defect masks and resamples only those regions.

Problem

Existing test-time scaling for text-to-image diffusion models resamples or searches the entire image, wasting compute on already-good regions and leaving localized defects insufficiently corrected.

Approach

Contrast cross-attention under "high-quality" vs "low-quality" prompt variants to get zero-shot quality-aware defect masks, then re-noise and denoise only masked regions at an intermediate timestep, followed by a global harmonization pass, inside a DFS best-of search.

Key results

• On SDXL (Pick-a-Pic), LoTTS reaches HPS 28.23 and IR 1.102 versus 24.54 HPS and 0.790 IR for Best-of-N.
• LoTTS achieves 2.75x-3.76x lower total wall-clock runtime than Best-of-N across SD2.1, SDXL and FLUX at comparable per-image latency.
• In a blind human study on SD2.1 with 100 prompts, LoTTS was preferred over Best-of-N in 86% of 500 judgments.

Surprising findings

• Random masks and removing the original-attention prior hurt performance, with the no-prior variant being the worst ablation.
• Removing the propagation step lowers scores only modestly on some metrics yet still reduces IR on all three models.

Limitations

• Failures on imperfect masks from subtle or high-frequency defects.
• Global defects (perspective, spatial relations, global semantics) cannot be fixed by local resampling.
• Low-quality initial images may be too fragile to refine, and near-perfect ones offer little room for improvement.

Experimental setup

Datasets: Pick-a-Pic DrawBench COCO2014

Benchmarks: Pick-a-Pic DrawBench COCO2014

Metrics: HPS v2 PickScore ImageReward Aesthetic Score FID CLIP score

Baselines: Resampling Best-of-N Particle Sampling

Builds on: SDEdit Best-of-N sampling Particle Sampling DPM-Solver++

Closest papers by meaning

• ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion (NeurIPS 2025 · similarity 0.90)
• On the Scalability of Diffusion-based Text-to-Image Generation (CVPR 2024 · similarity 0.89)
• Scaling Inference Time Compute for Diffusion Models (CVPR 2025 · similarity 0.89)
• SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer (ICML 2025 · similarity 0.89)
• Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.89)
• ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.88)