Research map / Image generation / Text-to-image diffusion

Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CePU, a consistency-enforced preference unlearning objective (C-DPO plus a consistency retention regularizer) that removes concepts directly from few-step distilled text-to-image models without re-distillation.

Problem

Preference-based unlearning (Diffusion-DPO and its unlearning variants) relies on noise-prediction errors and multi-step trajectories, so it transfers poorly to few-step distilled text-to-image diffusion models, and unlearning the base model then re-distilling is costly.

Approach

Replace the noise-prediction reward in DPO with a timestep-local consistency error (difference of x0 predictions at adjacent timesteps sharing the same noise), minimized for preferred and maximized for dispreferred samples, plus a consistency-based retention regularizer; a Jensen bound makes it single-step trainable.

Key results

• On DreamShaper-V8-LCM nudity unlearning, CePU reaches 95.83% average DSR with LPIPS 0.0834, versus 91.67% for DPO and 90.35% for DUO.
• On identity unlearning (DS-V8-LCM, Angelina Jolie), CePU gets target accuracy 0 with non-target accuracy 55.08 and harmonic mean 71.03, above SafetyDPO's 49.41.
• On Imagenette object unlearning, CePU reaches harmonic mean 75.66% and LPIPS 0.09, compared to FADE's 74.42% and 0.12.

Surprising findings

• Directly unlearning the FSD model performs comparably to unlearning the base model followed by re-distillation, at about 15 minutes on a single A5000 GPU.

Limitations

• The consistency surrogate reward is heuristic; the cross-timestep term is dropped and the sign-preservation claim holds only under a condition.
• Preference pairs are generated via SDEdit from the same models, so pair quality depends on the bootstrapped generator.
• Object-level unlearning drops the positive preference term and is not directly preference-based.

Experimental setup

Datasets: SDEdit-generated paired samples Imagenette

Benchmarks: I2P P4D MMA-Diffusion RAB SafeSora-Prompts (SP) Imagenette celebrity identity prompts

Metrics: target identity accuracy non-target accuracy harmonic mean LPIPS NudeNet defense success rate CLIP score

Baselines: DPO DUO PSO SafetyDPO ESD CA FADE UCE SAFREE

Builds on: Diffusion-DPO DUO SafetyDPO (AlignGuard) consistency models latent consistency models

Closest papers by meaning

• The Illusion of Unlearning: The Unstable Nature of Machine Unlearning in Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.92)
• Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning (NeurIPS 2025 · similarity 0.91)
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models (ICLR 2025 · similarity 0.90)
• Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models (CVPR 2025 · similarity 0.90)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient (AAAI 2025 · similarity 0.90)
• Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.90)