Research map / Image generation / Fast diffusion sampling

Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation

NeurIPS 2024 · Image generation › Fast diffusion sampling · approach: Distributional control of diffusion · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DUSA, which derives a single-timestep score-based identity linking diffusion noise predictions to the task model's posterior, and uses it to test-time adapt classifiers and segmenters via a diffusion model's conditional noise estimates.

Problem

Test-time adaptation of pre-trained discriminative models to out-of-distribution data using pre-trained diffusion models, where prior work (Diffusion-TTA) relies on costly Monte Carlo likelihood estimation over many timesteps and a low-dimensional conditioning space.

Approach

Score decomposition grad log p(x) = sum_y p(y|x) grad log p(x|y) holds at any single timestep, so the task model's posterior can be matched to the noise predictions of a conditional diffusion model at one timestep, with candidate-class selection (top-k plus multinomial sampling) and LogitNorm to reduce cost and stabilize training.

Key results

• On ImageNet-C fully TTA with ConvNeXt-L, DUSA reaches 66.2% average accuracy vs 61.1% for Diffusion-TTA (+5.1%).
• In continual TTA on ImageNet-C with ConvNeXt-L, DUSA reaches 67.7% vs 60.4% for Diffusion-TTA (+7.3%).
• On ADE20K-C with SegFormer-B5, DUSA reaches 28.1 mIoU vs 23.9 source and 23.7 for CoTTA.

Measured effects reported in the paper

Surprising findings

• Adapting the diffusion model is unnecessary: the unconditional-adaptation variant DUSA-U performs on par with DUSA.
• A single timestep (t=100) outperforms Diffusion-TTA with 6 sampled timesteps, which is a counter-intuitive efficiency gain.

Limitations

• Requires a diffusion model pre-trained on at least the same source data as the task model, which may not be readily available.
• Still slower than methods that only update the task model, so unsuitable for extreme efficiency demands such as autonomous driving.
• Pixelate corruption results fall below the source baseline without diffusion adaptation, indicating limited diffusion robustness to some OOD shifts.

Experimental setup

Datasets: ImageNet ImageNet-C ADE20K

Benchmarks: ImageNet-C ADE20K-C

Metrics: top-1 accuracy mIoU

Baselines: Tent CoTTA EATA SAR RoTTA Diffusion-TTA BN Adapt

Builds on: Diffusion-TTA Tweedie's formula LogitNorm Diffusion Classifiers CoTTA

Closest papers by meaning

• Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.87)
• Training Diffusion-based Generative Models with Limited Data (ICML 2025 · similarity 0.87)
• Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment (CVPR 2025 · similarity 0.87)
• Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models (NeurIPS 2025 · similarity 0.86)
• Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.86)
• Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.86)