Research map / Image generation / Fast diffusion sampling
DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates partially-labeled multi-task dense prediction as a joint diffusion-and-denoising problem over prediction or feature maps, with a Multi-Task Conditioning mechanism that uses all tasks' initial predictions to denoise each task.
Problem
Multi-task partially supervised learning (MTPSL) for dense scene understanding, where each training image is labeled for only a subset of tasks, yields noisy, low-quality predictions.
Approach
Applies DDPM-style forward noising and a learned cross-attention denoiser to initial multi-task prediction maps (Prediction Diffusion) or feature maps (Feature Diffusion), conditioned on a shared multi-task condition feature built from all tasks' initial outputs so unlabeled tasks also receive implicit supervision.
Key results
- On PASCAL one-label setting, DiffusionMTL (Prediction) improves MTL Perf Δm to +3.23% (ResNet-18) versus XTC* at +0.37%, with Feature Diffusion reaching +3.65%.
- On Cityscapes, DiffusionMTL (Feature) reaches 75.67 mIoU and 0.0130 absErr, versus XTC at 74.90 mIoU and 0.0161 absErr.
- DiffusionMTL uses 133M parameters versus 173M for XTC, with the diffusion denoiser adding 3.12 Δm points over an identical non-diffusion refinement model on ResNet-18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Prediction Diffusion | DiffusionMTL · PASCAL one-label | MTL Perf Δm: +3.23% | Helped |
| add Feature Diffusion | DiffusionMTL · PASCAL one-label | MTL Perf Δm: +3.65% | Helped |
| add Feature Diffusion | DiffusionMTL · PASCAL random-label | MTL Perf Δm: +4.27% | Helped |
| add Feature Diffusion | DiffusionMTL · NYUD random-label | MTL Perf Δm: -0.77% vs -3.44% | Helped |
| replace diffusion with iterative refinement (ablation) | DiffusionMTL (ResNet-18) · PASCAL one-label | MTL Perf Δm: +3.23% -> +0.11% | Hurt |
| replace diffusion with iterative refinement (ablation) | DiffusionMTL (ResNet-50) · PASCAL one-label | MTL Perf Δm: -0.66% -> -2.86% | Hurt |
| add Multi-Task Conditioning (ablation) | DiffusionMTL (ResNet-18) · PASCAL one-label | MTL Perf Δm: -1.33% -> +3.23% | Helped |
| add Multi-Task Conditioning (ablation) | DiffusionMTL (ResNet-50) · PASCAL one-label | MTL Perf Δm: -2.91% -> -0.66% | Helped |
| increase diffusion steps from one to two (ablation) | DiffusionMTL (Prediction) · Cityscapes | mIoU | Helped |
| increase diffusion steps (ablation) | DiffusionMTL (Prediction) · Cityscapes | FLOPs | Hurt |
Surprising findings
- On NYUD one-label setting, DiffusionMTL (Prediction) is worse than the MTL Baseline on Normal mErr (26.17 vs 26.44 is better, but Δm is -2.86% vs -3.99%), so gains are not uniform across all tasks and settings.
Limitations
- Performance gains increase with more diffusion steps but at higher FLOPs cost
- Feature Diffusion is more expensive (676G vs 628G FLOPs on PASCAL R18)
- Gains are not uniform: on NYUD Normal and some R50 settings, improvements are small or negative
Experimental setup
Datasets: PASCAL NYUD Cityscapes
Benchmarks: PASCAL NYUD Cityscapes
Metrics: mIoU absErr mErr maxF odsF MTL Perf Δm
Baselines: MTL Baseline SS XTC Single-Task
Builds on: DDPM XTC MTPSL
Closest papers by meaning
- Denoising Task Routing for Diffusion Models (ICLR 2024 · similarity 0.92)
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.90)
- Denoising Task Difficulty-based Curriculum for Training Diffusion Models (ICLR 2025 · similarity 0.89)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.88)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.