Research map / Vision-language and multimodal / Cross-modal learning
C3Net: Compound Conditioned ControlNet for Multimodal Content Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C3Net, a ControlNet-style any-to-any multimodal (image, text, audio) diffusion model that aligns modality conditions in CLIP space with unimodal-pretrained encoders and adds a trainable Control C3-UNet on top of CoDi's UNet.
Problem
Joint multimodal generation from compound conditions (image, text, audio) is limited: CoDi combines conditions only by linear interpolation in latent space, degrading synthesis quality, and multimodal paired datasets are scarce.
Approach
Align each modality to a shared CLIP latent space using encoders pretrained on unimodal data (MAE-based SSAST audio encoder) and then fine-tuned contrastively, and add a single trainable ControlNet-like Control C3-UNet copy that injects per-modality condition latents into the frozen CoDi C3-UNet skip connections, instead of relying only on latent interpolation.
Key results
- On the Tri-modality Test Set, C3Net improves image FID from 11.39 (CoDi) to 10.97 and CLIP score from 25.17 to 25.29.
- Text synthesis CIDEr-D rises from 0.0651 (CoDi) to 0.0713 on the Tri-modality Test Set.
- Audio classification accuracy on ESC-50 synthesized audio is 23.25% for C3Net vs 21.05% for CoDi; FAD is slightly worse (11.7 vs 11.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unimodal pre-training for alignment encoders instead of training the audio encoder from paired data only (ablation) | C3Net audio-conditioned text generation · AudioCap test set | CIDEr-D: 0.0654 -> 0.0704 | Helped |
| Use unimodal pre-training for alignment encoders instead of training the audio encoder from paired data only (ablation) | C3Net audio-conditioned text generation · AudioCap test set | SPIDEr: 0.0608 -> 0.0622 | Helped |
| Use unimodal pre-training for alignment encoders instead of training the audio encoder from paired data only (ablation) | C3Net audio-conditioned image generation · AudioSet test set | CLIP: 23.192 -> 23.325 | Helped |
| Use unimodal pre-training for alignment encoders instead of training the audio encoder from paired data only (ablation) | C3Net audio-conditioned image generation · AudioSet test set | Inception Score: 1.7730, 0.1450 -> 1.7732, 0.1535 | Mixed |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net image generation · Tri-modality Test Set | FID: 11.39 -> 10.97 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net image generation · Tri-modality Test Set | CLIP: 25.17 -> 25.29 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net text generation · Tri-modality Test Set | BLEU-1: 0.1059 -> 0.1104 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net text generation · Tri-modality Test Set | ROUGE-L: 0.1019 -> 0.1045 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net text generation · Tri-modality Test Set | CIDEr-D: 0.0651 -> 0.0713 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net text generation · Tri-modality Test Set | SPIDEr: 0.0631 -> 0.0665 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net audio generation · Tri-modality Test Set | OVL: 62.91 -> 63.25 | Helped |
| Add Control C3-UNet for compound conditioning rather than using CoDi's latent interpolation alone | C3Net audio generation · Tri-modality Test Set | FAD: 11.4 -> 11.7 | Hurt |
Surprising findings
- C3Net's audio FAD is slightly worse than CoDi's (11.7 vs 11.4), despite gains in OVL and REL.
- Gains over CoDi are small (e.g., FID 11.39 to 10.97, CLIP 25.17 to 25.29), and image inception scores are essentially tied (1.7730 vs 1.7732).
Limitations
- Shared CLIP latent space may not be optimal for audio.
- Contrastive alignment may discard modality-specific information.
- Tri-modality test set is constructed from AudioCaps with Stable Diffusion-generated images, and gains over CoDi are small.
Experimental setup
Datasets: AudioCaps AudioSet COCO LAION-400M Epidemic Sound effects ESC-50
Benchmarks: AudioCaps test set Tri-modality Test Set ESC-50 AudioSet test set
Metrics: CIDEr-D SPIDEr Inception Score CLIP score FID BLEU-1 ROUGE-L OVL REL FAD
Baselines: CoDi
Builds on: ControlNet CoDi CLIP SSAST
Closest papers by meaning
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.87)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.86)
- The Design Space of Tri-Modal Masked Diffusion Models (NeurIPS 2026 · similarity 0.86)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.85)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.