Research map / Vision-language and multimodal / Cross-modal learning
Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIVE, a semi-supervised 2D-3D cross-modal retrieval method that uses fine-grained prototypical voting for pseudo-labels and heterogeneous Mixup with soft similarity learning, optimized via sharpness-aware bi-level training.
Problem
Semi-supervised 2D-3D cross-modal retrieval, where only a few labeled 2D and 3D samples are available alongside many unlabeled ones, and the 2D and 3D modalities have a heterogeneous distribution gap.
Approach
Model multiple K-means fine-grained prototypes per class; each unlabeled sample retrieves its nearest M prototypes which vote for a pseudo-label. Cross-modal pairs with the same (pseudo-)label are mixed in feature space and aligned with a soft supervised-contrastive loss in a shared hidden space, all optimized with sharpness-aware minimization.
Key results
- FIVE reaches 93.50 average MAP on 3D MNIST 2D→3D versus 89.10 for ALGCN (second best).
- On ModelNet40 2D→3D average MAP is 72.94, compared with 61.86 for CLF.
- Ablation with 200 labels shows removing prototypical voting drops 3D MNIST I2P from 91.68 to 84.62.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add supervised learning for labeled 2D and 3D samples (ablation) | FIVE · 3D MNIST, 2D→3D, 200 labels | MAP: 19.47 -> 91.68 | Helped |
| Add supervised learning for labeled 2D and 3D samples (ablation) | FIVE · ModelNet10, 2D→3D, 200 labels | MAP: 19.71 -> 83.97 | Helped |
| Add supervised learning for labeled 2D and 3D samples (ablation) | FIVE · ModelNet40, 2D→3D, 800 labels | MAP: 5.39 -> 71.32 | Helped |
| Add fine-grained prototypical voting instead of single-centroid pseudo-labels (ablation) | FIVE · 3D MNIST, 2D→3D, 200 labels | MAP: 84.62 -> 91.68 | Helped |
| Add fine-grained prototypical voting instead of single-centroid pseudo-labels (ablation) | FIVE · ModelNet10, 2D→3D, 200 labels | MAP: 74.50 -> 83.97 | Helped |
| Add fine-grained prototypical voting instead of single-centroid pseudo-labels (ablation) | FIVE · ModelNet40, 2D→3D, 800 labels | MAP: 65.66 -> 71.32 | Helped |
| Add bi-level optimization rather than standard Adam (ablation) | FIVE · 3D MNIST, 2D→3D, 200 labels | MAP: 89.91 -> 91.68 | Helped |
| Add bi-level optimization rather than standard Adam (ablation) | FIVE · ModelNet10, 2D→3D, 200 labels | MAP: 76.95 -> 83.97 | Helped |
| Add bi-level optimization rather than standard Adam (ablation) | FIVE · ModelNet40, 2D→3D, 800 labels | MAP: 68.81 -> 71.32 | Helped |
| Add heterogeneous Mixup (ablation) | FIVE · 3D MNIST, 2D→3D, 200 labels | MAP: 90.91 -> 91.68 | Helped |
| Add heterogeneous Mixup (ablation) | FIVE · ModelNet10, 2D→3D, 200 labels | MAP: 82.18 -> 83.97 | Helped |
| Add heterogeneous Mixup (ablation) | FIVE · ModelNet40, 2D→3D, 800 labels | MAP: 70.34 -> 71.32 | Helped |
Surprising findings
- Removing supervised learning (FIVE w/o SL) collapses performance to around 5-19 MAP, showing the method depends heavily on the labeled signal.
Limitations
- Evaluated only on three 2D-3D datasets with rendered/image-point cloud pairs.
- Relies on fixed hyperparameters K and M selected on 3D MNIST, with sensitivity shown only there.
- Baselines from text-image retrieval were reproduced and adapted by the authors for 2D-3D setting.
Experimental setup
Datasets: 3D MNIST ModelNet10 ModelNet40
Benchmarks: 3D MNIST ModelNet10 ModelNet40
Metrics: mean average precision (MAP) precision-recall curves
Baselines: MRL DSCMR ALGCN DA-I-GCN DA-P-GCN DA-I-GAT DA-P-GAT CLF RONO
Builds on: CLF Mixup sharpness-aware minimization SupCon
Closest papers by meaning
- DREAM: Decoupled Discriminative Learning with Bigraph-aware Alignment for Semi-supervised 2D-3D Cross-modal Retrieval (AAAI 2025 · similarity 0.82)
- NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval (CVPR 2025 · similarity 0.80)
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.80)
- Cross-Modal 3D Representation with Multi-View Images and Point Clouds (CVPR 2025 · similarity 0.80)
- Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (NeurIPS 2025 · similarity 0.80)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.