Research map / Image generation / Text-to-image diffusion

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

ICLR 2025 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Diff-2-in-1, an integrated latent diffusion (Stable Diffusion) framework that performs dense perception and multi-modal (RGB + annotation) data generation, using in-distribution partial-noise synthesis and an EMA-based self-improving loop between creation and exploitation parameters.

Problem

Diffusion models are usually used as standalone components for dense perception (augmentation or feature extraction), and synthetic data from Gaussian noise has a domain gap that limits its usefulness for discriminative training.

Approach

A single diffusion model both predicts pixel-level labels from RGB and generates paired RGB-label data; partial-noise (halfway) denoising keeps synthetic samples in-distribution, and an EMA-updated creation network is refined by feedback from the exploitation network in a self-improving cycle.

Key results

• On NYUv2 surface normal with 795 samples, Diff-2-in-1 on iDisc reaches 68.7 at 11.25° and 12.7 mean error vs 57.3 and 17.8 for iDisc trained on 795 samples alone.
• On ADE20K, Diff-2-in-1 on VPD improves mIoU from 53.7 to 54.5.
• On NYUD-MT, Diff-2-in-1 on TaskPrompter reaches 55.73 semseg mIoU and 17.91 normal mErr versus 55.30 and 18.47 for the baseline.

Measured effects reported in the paper

Surprising findings

• Training on ScanNet with Diff-2-in-1 (63.0 at 11.25°) outperforms the baseline trained on NYUv2 (62.2), suggesting cross-domain generalization.
• Using generated data from the self-improving pipeline matches the performance of using 500 additional real unlabeled NYUv2 images.

Limitations

• Frozen denoising U-Net in most settings limits the model's adaptation; finetuning the backbone (direct or LoRA) hurt performance.
• Requires an image captioning model (BLIP-2) to provide text prompts; without captions performance drops noticeably.

Experimental setup

Datasets: NYUv2 ADE20K NYUD-MT PASCAL-Context ScanNet Tiny-Taskonomy

Benchmarks: NYUv2 ADE20K NYUD-MT PASCAL-Context Tiny-Taskonomy

Metrics: mIoU angle error accuracy (11.25/22.5/30 deg) mean/median angle error RMSE maxF mErr Top-1 accuracy

Baselines: VPD DA-Fusion Gaussian Noise Augmentation Marigold iDisc InvPT TaskPrompter Bae et al. (NYUv2 surface normal)

Builds on: Latent diffusion models VPD mean teacher SDEdit DA-Fusion

Closest papers by meaning

• Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception (ICLR 2025 · similarity 0.92)
• What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.92)
• Scaling Properties of Diffusion Models For Perceptual Tasks (CVPR 2025 · similarity 0.90)
• DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.90)
• DiffBEV: Conditional Diffusion Model for Bird’s Eye View Perception (AAAI 2024 · similarity 0.90)
• Why DiT Models Underperform as Representation Learners without Long Skip Connections (NeurIPS 2026 · similarity 0.89)