Research map / Label-efficient and robust learning / Semi-supervised learning

Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities

AAAI 2025 · Label-efficient and robust learning › Semi-supervised learning · approach: Confidence-based pseudo-label selection · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MSC, a semi-supervised multimodal (image-text) classification framework that learns modal reliability weights and dual-KL strategic guidance between score-fusion and feature-concat pseudo-labeling branches, built on FixMatch/FreeMatch.

Problem

Multimodal image-text classification is heavily dependent on abundant labeled data; the paper targets semi-supervised multimodal classification (SSMC) with only a few labels per class plus unlabeled pairs.

Approach

Exploits modal complementarity (learned instance-level reliability weights for text and image predictions, with KL guidance from the more reliable modality) and strategic complementarity (consensus-based pseudo-label selection and dual KL mutual guidance between Score Fusion and Feature Concat branches) on top of FixMatch/FreeMatch.

Key results

• FixMatch+MSC reaches 91.08% accuracy on UPMC-Food101 with 20 labels per class, versus 88.81 for supervised Score Fusion and 86.65 for FixMatch+FC (approx. 4-5 points over best baselines).
• FreeMatch+MSC reaches 76.98% accuracy on N24News with 20 labels per class, above the best baseline around 76.3.
• On CrisisMMD with 100 labels per class, FixMatch+MSC reaches 88.00% accuracy, close to the latest supervised methods (within about 1-5 points).

Measured effects reported in the paper

Surprising findings

• Simple learned modal weighting without guidance signals (simple weight) performs worst on N24News (69.77) and UPMC-Food101 (80.44), below the base methods.
• Removing the modal reliability guidance module drops UPMC-Food101 accuracy sharply (to 73.08 from 91.08), while LCG and SCG matter more on N24News.

Limitations

• Weighted Score Fusion selects pseudo-labels similarly to averaged Score Fusion, so the modal weighting does not fully resolve the quantity-quality trade-off by itself.
• Results vary notably across datasets; module contributions differ between N24News and UPMC-Food101.
• Many hyperparameters (loss weights, threshold eta=0.95) must be tuned per dataset.

Experimental setup

Datasets: N24News UPMC-Food101 CrisisMMD

Benchmarks: N24News UPMC-Food101 CrisisMMD

Metrics: accuracy F1

Baselines: BERT ViT Score Fusion Feature Concat FixMatch FreeMatch UniS-MMC CMA-CLIP MMBT finetuned CLIP DMCC SSE-Cross

Builds on: FixMatch FreeMatch Score Fusion Feature Concat

Closest papers by meaning

• STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classification (CVPR 2025 · similarity 0.89)
• Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.89)
• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
• Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.87)
• Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.87)
• Seek Common Ground While Reserving Differences: Semi-Supervised Image-Text Sentiment Recognition (CVPR 2025 · similarity 0.86)