Research map / Image generation / Autoregressive and latent generation
Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDDBM, a latent denoising diffusion bridge that translates between arbitrary modalities via a shared latent space, with contrastive alignment and predictive losses, evaluated on multiview-to-3D shape, zero-shot super-resolution, and other MT tasks.
Problem
Diffusion bridge models (DDBMs) assume source and target share the same dimensionality and space, so they cannot translate across heterogeneous modalities (e.g., multi-view images to 3D shapes, low- to high-resolution images) without modality-specific architectures.
Approach
Lift the DDBM bridge into a shared latent space using simple modality-specific encoders/decoders, a domain-agnostic encoder-decoder transformer denoiser with cross-attention and a learnable MASK token, plus a predictive end-to-end loss and an InfoNCE contrastive alignment loss, trained iteratively.
Key results
- On ShapeNet multiview-to-3D generation, LDDBM reaches 1-NNA .508 and IoU .664, beating EDM (.532 / .631), DiT, and SiT.
- Zero-shot 16x16 to 128x128 super-resolution on CelebA-HQ: PSNR 25.6, SSIM 0.68, LPIPS 0.32, versus DiWa's baseline.
- On edges-to-bags, LDDBM gets FID 4.17 vs DDBM 2.93 but runs in 7.8 s vs 16.9 s, over 2x faster inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace U-Net backbone with DiT (ablation) | LDDBM architecture · ShapeNet | IoU: .635 -> .613 | Hurt |
| Replace U-Net backbone with DiT (ablation) | LDDBM architecture · CelebA-HQ | PSNR: 23.2 -> 22.2 | Hurt |
| Add encoder-decoder design to DiT (ablation) | LDDBM architecture · ShapeNet | IoU: .613 -> .651 | Helped |
| Add spatial embeddings (ablation) | LDDBM architecture · ShapeNet | IoU: .651 -> .658 | Helped |
| Add spatial embeddings (ablation) | LDDBM architecture · CelebA-HQ | PSNR: 23.4 -> 22.9 | Hurt |
| Add learnable [MASK] token (ablation) | LDDBM architecture · CelebA-HQ | PSNR: 22.9 -> 25.6 | Helped |
| Use predictive loss instead of separate reconstruction losses (ablation) | LDDBM · ShapeNet | IoU: .609 ± .007 -> .643 ± .005 | Helped |
| Add contrastive loss to predictive loss (ablation) | LDDBM · ShapeNet | IoU: .643 ± .005 -> .664 ± .002 | Helped |
| Add contrastive loss to predictive loss (ablation) | LDDBM · CelebA-HQ | LPIPS: 0.41 ± .02 -> 0.32 ± .01 | Helped |
| Use iterative training instead of two-step training (ablation) | LDDBM · ShapeNet | IoU: .637 ± .003 -> .664 ± .002 | Helped |
| Use iterative training instead of end-to-end training (ablation) | LDDBM · CelebA-HQ | PSNR: 23.4 ± 0.3 -> 25.6 ± 0.4 | Helped |
| Reduce sampling steps from 40 to 30 (ablation) | LDDBM · ShapeNet | 1-NNA: 508 ± .005 -> 504 ± .006 | Helped |
Surprising findings
- On edges-to-bags, the general LDDBM has worse FID than DDBM (4.17 vs 2.93) despite being faster.
- Pre-trained versus vanilla autoencoder gave near-identical PSNR (25.6 vs 25.3), suggesting limited benefit from pre-training.
Limitations
- Reported results use a single seed-scale setup on small benchmarks; the 1-NNA on nuScenes-Occupancy remains high (~0.81), showing a large gap to realistic generation.
- Translation is restricted to paired data; unpaired translation is left to future work.
- Encoder/decoder quality strongly affects performance; a simple CNN substantially underperforms (PSNR 22.9).
Experimental setup
Datasets: ShapeNet nuScenes-Occupancy FFHQ CelebA-HQ DIV2K edges2handbags (DDBM benchmark)
Benchmarks: ShapeNet nuScenes-Occupancy CelebA-HQ DIV2K edges2handbags
Metrics: 1-NNA IoU PSNR SSIM LPIPS FID inference time
Baselines: EDM 3D-EDM DiT SiT Pix2Vox-A DiWa DDBM
Builds on: Denoising Diffusion Bridge Models DiT CLIP Latent Diffusion Models
Closest papers by meaning
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution (CVPR 2025 · similarity 0.88)
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.88)
- SynthRGB-T: Language-Vision Guided Image Translation for Diversity Synthesis (CVPR 2026 · similarity 0.88)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.