Research map / Vision-language and multimodal / Cross-modal learning
Super Deep Contrastive Information Bottleneck for Multi-modal Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDCIB, a variational information-bottleneck multi-view clustering method that injects all encoder hidden-layer information and feature- and cluster-level cross-modal consistency into the loss, evaluated on four multi-modal image datasets.
Problem
Multi-modal clustering struggles to deeply explore latent information and interdependencies between modalities, as existing methods mostly fuse at the output layer and consider either feature or cluster consistency, not both.
Approach
Explicitly includes mutual information terms for all encoder hidden layers (compression against inputs, relevance to cluster assignments, and cross-modality consistency of hidden features) plus cluster-assignment consistency across modalities, estimated with variational bounds, InfoNCE and MINE.
Key results
- SDCIB reaches 67.5% ACC and 59.2% NMI on Caltech-2V, versus 64.1% ACC for the best prior method DIVICE.
- On ESP, SDCIB achieves 61.4% ACC and 44.7% NMI, a 9.3-point ACC gain over the best compared method.
- Removing the hidden-layer consistency term drops ACC on Event from 56.5% to 45.0%, and removing both consistency terms drops Caltech-2V ACC to 32.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SDCIB instead of the strongest compared baseline | multi-modal clustering · Caltech-2V | ACC: 64.1 -> 67.5 | Helped |
| add SDCIB instead of the strongest compared baseline | multi-modal clustering · Event | ACC: 49.1 -> 56.5 | Helped |
| add SDCIB instead of the strongest compared baseline | multi-modal clustering · IAPR | ACC: 47.3 -> 52.9 | Helped |
| add SDCIB instead of the strongest compared baseline | multi-modal clustering · ESP | ACC: 52.1 -> 61.4 | Helped |
| add clustering-assignment consistency I(Yᵢ;Yⱼ) (ablation) | SDCIB · Caltech-2V | ACC: 41.8 -> 67.5 | Helped |
| add clustering-assignment consistency I(Yᵢ;Yⱼ) (ablation) | SDCIB · Event | ACC: 30.3 -> 56.5 | Helped |
| add hidden-layer feature consistency I(Hᵢₗ;Hⱼₗ) (ablation) | SDCIB · Caltech-2V | ACC: 63.1 -> 67.5 | Helped |
| add hidden-layer feature consistency I(Hᵢₗ;Hⱼₗ) (ablation) | SDCIB · Event | ACC: 45.0 -> 56.5 | Helped |
| add both clustering-assignment and hidden-layer feature consistency (ablation) | SDCIB · Caltech-2V | ACC: 32.2 -> 67.5 | Helped |
| add both clustering-assignment and hidden-layer feature consistency (ablation) | SDCIB · Event | ACC: 21.6 -> 56.5 | Helped |
| add encoder hidden-layer information rather than using only final output HᵢL (ablation) | SDCIB · Caltech-2V | ACC: 63.6 -> 67.5 | Helped |
| add encoder hidden-layer information rather than using only final output HᵢL (ablation) | SDCIB · Event | ACC: 45.9 -> 56.5 | Helped |
Surprising findings
- Removing the hidden-layer consistency term slightly increases ESP ACC (62.3 vs 61.4), an exception to the ablation trend.
- Parameter analysis reports the best results at alpha=0.1 and beta=1, while the text also says the loss converges around 500 epochs despite training for 40 epochs.
Limitations
- Limited performance on incomplete data samples.
- Requires a predetermined number of clusters.
- Uses a batch learning strategy, limiting real-time use.
Experimental setup
Datasets: Caltech-2V Event IAPR ESP
Benchmarks: Caltech-2V Event IAPR ESP
Metrics: ACC NMI
Baselines: KM Ncuts AmKM AmNcuts CoregMVSC RMKMC SwMC ONMSC SiMVC CoMVC MFLVC DealMVC
Builds on: information bottleneck InfoNCE MINE DIVICE MFLVC
Closest papers by meaning
- Multi-aspect Self-guided Deep Information Bottleneck for Multi-modal Clustering (AAAI 2025 · similarity 0.92)
- A Peer-review Look on Multi-modal Clustering: An Information Bottleneck Realization Method (ICML 2025 · similarity 0.88)
- Differentiable Information Bottleneck for Deterministic Multi-view Clustering (CVPR 2024 · similarity 0.86)
- GeCC: Generalized Contrastive Clustering with Domain Shifts Modeling (AAAI 2025 · similarity 0.85)
- Multi-Level Cross-Modal Alignment for Image Clustering (AAAI 2024 · similarity 0.85)
- Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.