Research map / Image generation / Text-to-image diffusion

Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dual-FDM, a training-free frequency-band masking method in cross-attention of IP-Adapter/SDXL that disentangles customization and color/style references for personalized image generation, with entropy-based adaptive early/late timestep modulation.

Problem

Personalized image generation with dual image references (customization and style/color) suffers from text misalignment: foreground/background entanglement across frequency bands overwhelms text prompt guidance during early denoising.

Approach

Splits reference-image key/value into a foreground matrix (W_k) and a background matrix via inverted singular values (1/sigma), then uses DCT-domain masks to swap mid-frequency (customization) or low-frequency (color) bands between references, with timestep switching chosen by an information-entropy ratio.

Key results

• Dual-FDM_cust reports CLIP-T 0.3194 and CLIP-TC 0.5031 on the CSGO setting, versus 0.3123 and 0.4802 for the next best (MS-Diffusion+CSGO and IP-Adapter respectively).
• Dual-FDM_cust reports Image Reward 5.3487 versus 4.2023 for the second-best method in the CSGO comparison table.
• Dual-FDM_cost achieves CLIP-TS 0.5015 and DINO-S 0.4970 on the color style transfer setting, above StyleAligned (0.4832 CLIP-TS).

Surprising findings

• Inverting singular values (1/sigma) of the key projection, rather than hard-zeroing top singular components, is reported to give more stable background disentanglement across images.

Limitations

• Relies on IP-Adapter with SDXL and hand-set frequency thresholds (th_mp1, th_mp2, th_lp).
• Limited to two-reference setting (customization plus style or color).

Experimental setup

Datasets: DreamBench TI benchmark DreamBench++ IS-Plus StyleBench T2I-CompBench++

Benchmarks: DreamBench TI benchmark DreamBench++ IS-Plus StyleBench T2I-CompBench++

Metrics: CLIP-T CLIP-TC CLIP-TS CLIP-CS CLIP-S DINO-S Image Reward HPS Color L2 Color L1

Baselines: IP-Adapter SSR-Encoder MIP-Adapter Dreamcache PatchDPO MS-Diffusion CSGO StyleID StyleSSP InstantStyle StyleShot StyleAligned

Builds on: IP-Adapter Stable Diffusion XL MS-Diffusion StyleStudio

Closest papers by meaning

• Free-Lunch Color-Texture Disentanglement for Stylized Image Generation (NeurIPS 2025 · similarity 0.88)
• One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.88)
• FDS: Frequency-Aware Denoising Score for Text-Guided Latent Diffusion Image Editing (CVPR 2025 · similarity 0.88)
• Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.87)
• Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models (ICLR 2025 · similarity 0.87)
• Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization (ICLR 2025 · similarity 0.87)