Research map / Image generation / Text-to-image diffusion

Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-Aid, a lightweight plug-in module that adaptively modulates per-token text features across transformer blocks and denoising timesteps in frozen FLUX and SD 3.5 models, trained with diffusion, DPO and regularization losses.

Problem

Text-to-image diffusion transformers often fail to faithfully follow complex prompts due to insufficient interaction between text and image tokens; prior fixes use static CFG, handcrafted token/block weighting, or architectural changes that require retraining.

Approach

A small Adaptive Interaction Denoising (AiD) module predicts per-token, block- and timestep-dependent modulation coefficients in [-1,1] for text features via a tanh-gated MLP, trained on a frozen backbone with diffusion + DPO losses, and applied at inference as a plug-in.

Key results

• FLUX + Diff-Aid reaches 10.71 overall HPSv3 vs 10.42 for FLUX baseline, slightly above Kolors (10.55).
• SD 3.5 + Diff-Aid improves GenEval from 0.72 to 0.77 and ImageReward from 0.94 to 1.05.
• Applied zero-shot to FLUX.1 Kontext on EditBench, CLIP-I rises by 0.93 (90.74 to 91.67) and CLIP-T by 0.06.

Surprising findings

• Learned modulation coefficients show abrupt changes at timesteps coinciding with shifts in attention norms, and a clear drop in single-stream FLUX blocks at t=0.607.
• Simply clipping the modulation values instead of tanh bounding degrades HPSv3 sharply (7.12 vs 10.61), and removing bounding entirely collapses performance.

Limitations

• Only evaluated on rectified text-to-image DiT models; not extended to video or 3D generation.
• Gains on some metrics are marginal (e.g., LoRA setting ImageReward drops from 0.89 to 0.86 and FLUX + Ours GenEval 0.70 vs baseline 0.68 is small).

Experimental setup

Datasets: HPDv3 HPDv3 benchmark

Benchmarks: HPSv3 benchmark HPSv2 ImageReward Aesthetic Score GenEval EditBench

Metrics: HPSv3 HPSv2 ImageReward Aesthetic Score GenEval accuracy CLIP-I CLIP-T

Baselines: TACA Kolors FLUX SD 3.5 Infinity SD 3 LoRA + DPO FLUX.1 Kontext

Builds on: DPO classifier-free guidance MMDiT TACA

Closest papers by meaning

• One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.90)
• Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
• T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
• Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning (NeurIPS 2025 · similarity 0.88)
• Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.88)
• PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.87)