Research map / Image generation / Text-to-image diffusion
JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends a pre-trained text-to-image diffusion model into a joint RGB-plus-dense-modality (RGBD) generator by adding a trainable, densely connected copy of the network while the RGB branch stays frozen.
Problem
Modeling the joint distribution of images and dense modalities (e.g., depth, normals) is hard because labeled dense data is far smaller than text-image corpora, and naive extension of diffusion models causes catastrophic forgetting of RGB generation quality.
Approach
A ControlNet-inspired dual-branch design: a trainable copy of the pre-trained diffusion UNet handles the dense modality, bidirectionally connected to the frozen RGB branch via zero-initialized convolutions, so the initial output equals the original model (output preserving principle). Downstream uses include channel-wise inpainting for bidirectional dense prediction and tile-based 3D panorama gene
Key results
- Joint RGBD generation FID 14.38 (depth) vs 16.15 for SD v2.1 and 32.13 for LDM3D, with CLIP 25.71 vs 26.17 for SD v2.1.
- Tile-based panorama generation achieves intra-LPIPS 0.578 at 17 s, versus 0.641 for Multi-Diffusion (49 s) and 0.626 for Sync-Diffusion (101 s).
- Mono-view depth estimation is comparable to MiDaS in RMSE but worse in AbsRel.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Extend Stable Diffusion v2.1 with JointNet for depth generation instead of Direct Extend | Stable Diffusion v2.1 · MSCOCO validation prompts | FID: 14.15 -> 14.38 | Hurt |
| Extend Stable Diffusion v2.1 with JointNet for depth generation instead of Direct Extend | Stable Diffusion v2.1 · MSCOCO validation prompts | IS: 37.08 -> 38.33 | Helped |
| Extend Stable Diffusion v2.1 with JointNet for depth generation instead of Direct Extend | Stable Diffusion v2.1 · MSCOCO validation prompts | CLIP: 25.74 -> 25.71 | Hurt |
| Use random offset, full-image denoising, and boundary decay in tile-based generation | JointNet panorama generation · RGBD panorama generation | Intra-LPIPS: 0.641 -> 0.578 | Helped |
| Use random offset, full-image denoising, and boundary decay in tile-based generation | JointNet panorama generation · RGBD panorama generation | Time (s): 49 -> 17 | Helped |
| Use JointNet rather than Direct Extend for joint RGBD generation | Stable Diffusion v2.1 · RGB generation during joint generation | RGB generation capability | Helped |
Surprising findings
- Direct extension of the first and last layers causes the model to lose RGB generation ability in early training steps before partially recovering.
- The proposed method does not beat Direct Extend on FID despite qualitative improvements, which the authors attribute to FID's inability to capture detail differences.
Limitations
- Inference time is roughly doubled because two diffusion branches are kept.
- Scaling to more modalities requires more copies, increasing cost.
- Depth AbsRel is worse than MiDaS, and the model is only trained on depth and normal modalities with RGB frozen in the first stage.
Experimental setup
Datasets: COYO-700M MSCOCO validation MegaDepth
Benchmarks: MSCOCO validation (30K prompts) MegaDepth
Metrics: FID IS CLIP similarity AbsRel RMSE intra-LPIPS running time
Baselines: LDM3D Direct Extend Stable Diffusion v2.1 MiDaS Multi-Diffusion Sync-Diffusion depth-conditioned Stable Diffusion v2 ControlNet
Builds on: ControlNet Stable Diffusion MiDaS SDEdit Multi-Diffusion
Closest papers by meaning
- Exploiting Diffusion Prior for Generalizable Dense Prediction (CVPR 2024 · similarity 0.86)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.86)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.85)
- Neural Residual Diffusion Models for Deep Scalable Vision Generation (NeurIPS 2024 · similarity 0.85)
- Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction (ICLR 2025 · similarity 0.85)
- Taming Stable Diffusion for Text to 360 Panorama Image Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.