Research map / 3D vision / 3D generation
Rethinking Multi-view Representation Learning via Distilled Disentangling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRDD, a two-stage multi-view representation learning framework that learns view-consistent representations via masked cross-view prediction and then distills view-specific representations by minimizing a CLUB mutual-information upper bound against a frozen consistent encoder.
Problem
Multi-view representation learning struggles with redundancy between view-consistent and view-specific representations, and joint end-to-end min-max optimization of mutual information bounds often yields suboptimal solutions.
Approach
Masked cross-view prediction trains a single consistent VAE-style encoder to predict all masked views from visible blocks; then a frozen consistent encoder is used to filter consistency information out of per-view specific encoders via CLUB-based MI minimization plus reconstruction from concatenated [s, c] codes.
Key results
- MRDD-cs reaches 75.93% clustering ACC on E-MNIST vs 67.10% for the best baseline GCFAgg (+8.83).
- On E-MNIST classification, MRDD-cs achieves 98.37% accuracy vs 94.21% for UNITER.
- Ablation shows only Stage I gives 93.69% and only Stage II gives 62.13% on E-MNIST classification, supporting the two-stage design.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MRDD-cs instead of the strongest reported baseline | MRDD-cs · E-MNIST, E-FMNIST, COIL-20, COIL-100, Office-31 | ACCclu and NMI: +8.83; +7.66; +4.54; +4.86; +1.64; +0.91; +4.98; +3.34; +2.86; +9.04 | Mixed |
| Add MCP (ablation) | MRDD-cs · E-MNIST classification | ACCcls: 93.12 -> 98.37 | Helped |
| Add Ld (ablation) | MRDD-cs · E-MNIST classification | ACCcls: 94.57 -> 98.37 | Helped |
| Add Lr (ablation) | MRDD-cs · E-MNIST classification | ACCcls: 90.63 -> 98.37 | Helped |
| Use both stages rather than stage I only (ablation) | MRDD-cs · E-MNIST classification | ACCcls: 93.69 -> 98.37 | Helped |
| Use both stages rather than stage II only (ablation) | MRDD-cs · E-MNIST classification | ACCcls: 62.13 -> 98.37 | Helped |
| Increase the MCP masked ratio; performance generally improves, with most datasets peaking at 70% (ablation) | MRDD-cs · Five datasets | ACCcls | Helped |
| Use random masking rather than block-wise or grid-wise masking (ablation) | MRDD-cs · Five datasets | ACCcls: 98.37; 88.78; 95.97; 91.17; 73.51 | Helped |
| Reduce consistency representation dimension relative to specificity dimension (ablation) | MRDD-cs · E-MNIST and E-FMNIST | ACCclu | Helped |
| Concatenate all representations instead of using c + s1 (ablation) | MRDD-cs · Five datasets | ACCclu: 75.93; 58.25; 69.18; 62.00; 37.14 -> 73.16; 58.10; 63.26; 60.90; 40.84 | Hurt |
Surprising findings
- Simply concatenating all consistent and specific representations degrades performance due to redundancy.
- Consistent-only representations (MRDD-c) outperform the concatenated version on NMI and clustering for several datasets.
Limitations
- Effectiveness of disentangling depends on quality of consistent representations.
- MCP increases training instability; reconstructing original data is hard in complex scenarios.
- Experiments limited to five relatively small datasets, mostly synthetic multi-view constructions.
Experimental setup
Datasets: E-MNIST E-FMNIST COIL-20 COIL-100 Office-31
Benchmarks: E-MNIST E-FMNIST COIL-20 COIL-100 Office-31
Metrics: clustering ACC NMI classification ACC F-Score
Baselines: Joint-VAE β-VAE MFLVC CONAN CMC EAMC GCFAgg Multi-VAE MIB DVIB UNITER
Builds on: CLUB mutual information estimator beta-VAE masked autoencoders variational information bottleneck
Closest papers by meaning
- Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs (ICLR 2025 · similarity 0.87)
- Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning (AAAI 2024 · similarity 0.87)
- Multi-View Causal Representation Learning with Partial Observability (ICLR 2024 · similarity 0.87)
- CFDM: Contrastive Fusion and Disambiguation for Multi-View Partial-Label Learning (AAAI 2025 · similarity 0.87)
- Incomplete Multi-View Multi-label Learning via Disentangled Representation and Label Semantic Embedding (CVPR 2025 · similarity 0.87)
- Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.