Research map / Vision-language and multimodal / Cross-modal learning
Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Probabilistic Conformal Distillation (PCD), a self-distillation method that models each modality-missing feature as a Gaussian distribution and fits it to the unknown PDF of modality-complete features, improving missing-modality robustness on face anti-spoofing and RGB-D segmentation benchmarks.
Problem
Multimodal models trained on modality-complete data degrade severely when some modalities are missing at test time; deterministic cross-modal knowledge distillation that point-aligns missing-modality features to complete-modality features is too stringent given information asymmetry.
Approach
Replace point alignment in cross-modal distillation with a probabilistic objective: each modality-missing input is a learned Gaussian whose log-likelihood is maximized at positive complete-modality features and minimized at negatives, plus a contrastive term that aligns the geometric (distance-based) structure of peak points with the teacher's.
Key results
- On CeFA, PCD reaches 22.63% average ACER versus 27.94% for the second-best method MMANET/ETMC-level baseline (about 5.31% reduction), with a 5.31% average improvement reported over the second-best.
- On CASIA-SURF, PCD achieves 2.93% average ACER, a 1.01% reduction over the second-best method (MMANET at 3.94%).
- On Cityscapes and NYUv2, PCD improves average mIOU by 0.83% and 0.91% over the second-best method, reaching 73.03 and 46.49.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use probabilistic conformal distillation instead of the strongest reported baseline | multimodal models · CASIA-SURF (Average) | ACER: 1.01%↓ | Helped |
| use probabilistic conformal distillation instead of the strongest reported baseline | multimodal models · CeFA (Average) | ACER: 5.31%↓ | Helped |
| use probabilistic conformal distillation instead of the strongest reported baseline | multimodal segmentation models · NYUv2 (Average) | mIOU: 0.91% | Helped |
| use probabilistic conformal distillation instead of the strongest reported baseline | multimodal segmentation models · Cityscapes (Average) | mIOU: 0.83% | Helped |
| replace determinate Euclidean alignment with probabilistic distillation (ablation) | PCD on CeFA · CeFA (Average) | ACER: 8.36% on average | Helped |
| use self-KD instead of a separately pretrained teacher (ablation) | PCD on CeFA · CeFA (Average) | ACER: 5.71% lower on average | Helped |
| add probability extremum loss (ablation) | classification model with classification loss on CeFA · CeFA (Average) | ACER: 3.36% | Helped |
| add geometric consistency loss to classification and probability extremum losses (ablation) | PCD on CeFA · CeFA (Average) | ACER | Helped |
| add probability extremum loss (ablation) | classification-only model · CASIA-SURF (Average) | ACER: 6.78 -> 4.05 | Helped |
| add geometric consistency loss (ablation) | classification-only model · NYUv2 (Average) | mIOU: 44.43 -> 45.82 | Helped |
| add probability extremum and geometric consistency losses (ablation) | classification-only model · Cityscapes (Average) | mIOU: 71.89 -> 73.03 | Helped |
| vary the warm-up duration from 30 to 80 epochs (ablation) | PCD on CeFA · CeFA (Average) | ACER: 23.98; 23.08; 22.63; 23.01; 23.86; 23.74 | Mixed |
Surprising findings
- Some existing methods get worse when more modalities are present (e.g., MMANET on CeFA is 0.81% worse with complete modalities than with IR missing), which the authors attribute to overfitting from deterministic alignment.
Limitations
- Designed only for modality-complete training data; distillation cannot be applied to modality-missing training samples, though an extension is tested.
- Standard deviations are reported only in the appendix stability experiments; the main results do not include error bars.
- Gains on depth-missing segmentation scenarios are small because RGB performance is already close to the complete-input performance.
Experimental setup
Datasets: CASIA-SURF CeFA NYUv2 Cityscapes SUN RGB-D
Benchmarks: CASIA-SURF CeFA NYUv2 Cityscapes SUN RGB-D
Metrics: ACER mIOU
Baselines: Traditional Separate Model Augmentation HeMIS MMFormer MMANET MD ETMC RAML
Builds on: cross-modal knowledge distillation self-KD supervised contrastive learning ESANet ETMC
Closest papers by meaning
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.89)
- Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · similarity 0.87)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
- RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates (AAAI 2024 · similarity 0.87)
- Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.86)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.