Research map / Image generation / Text-to-image diffusion

Consistency-Preserving Concept Erasure via Unsafe–Safe Pairing and Directional Fisher-weighted Adaptation

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Concept erasure in diffusion models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAIR, a concept-erasure method for text-to-image diffusion that uses synthetically generated unsafe–safe image pairs for paired semantic realignment and FiDoRA, a Fisher-weighted directional DoRA initialization, achieving strong erasure and consistency with few trainable parameters.

Problem

Existing concept erasure methods in text-to-image diffusion models map target concepts to a generic null space, which damages structural and semantic consistency of related outputs and typically requires full fine-tuning that is computationally costly.

Approach

Replace the null-space target in ESD with a semantically aligned safe counterpart (image and text) built from unsafe–safe pairs produced by image generation and instruction-based editing, and initialize DoRA low-rank directional adapters via a Fisher-weighted SVD that targets forget-set-sensitive directions while staying low-sensitivity on retain-set data.

Key results

• On nudity removal (I2P target prompts), PAIR reaches ASR-IG 2.67 and the highest harmonic mean (66.83) versus 64.66 for the next-best method (RECE).
• On Van Gogh style removal, PAIR achieves FID 16.90 and HM 63.07, the best among compared methods.
• Human evaluation win rate of 49.2% and GPT-5.2 judge win rate of 74.0% favor PAIR over ESD, RECE, and AGE.

Surprising findings

• SPM reports the highest consistency score on nudity removal but has a high ASR-IG (25.00), so the consistency metric is uninformative for failed erasure.

Limitations

• Relies on classifier-filtered synthetic unsafe-safe pairs whose quality depends on the editing models (RealEdit, ICEdit).
• Human evaluation uses only 50 participants and moderate inter-rater agreement (Fleiss' kappa 0.462).

Experimental setup

Datasets: I2P COCO-10K MMA-Diffusion Ring-A-Bell

Benchmarks: I2P MMA-Diffusion Ring-A-Bell UnlearnDiff COCO-10K

Metrics: ASR FID CLIP score SSIM DINO score AugCLIP CFD harmonic mean

Baselines: SD v1.4 ESD SPM RECE Co-Erasing AGE

Builds on: ESD DoRA FILA IP-Adapter LoRA

Closest papers by meaning

• Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models (NeurIPS 2025 · similarity 0.90)
• MACE: Mass Concept Erasure in Diffusion Models (CVPR 2024 · similarity 0.90)
• Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters (CVPR 2025 · similarity 0.89)
• Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation (CVPR 2025 · similarity 0.89)
• Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them (ICLR 2025 · similarity 0.89)
• All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)