Research map / Vision-language and multimodal / Cross-modal learning
Fuzzy Multimodal Learning for Trusted Cross-modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FUME, a fuzzy-set-based trusted cross-modal retrieval framework that estimates epistemic uncertainty from category credibility (possibility plus necessity) and merges per-modality entropy-based uncertainties to filter unreliable retrievals.
Problem
Deterministic cross-modal retrieval models return similarity-ranked results without quantifying uncertainty, so unreliable ("incredible") matches go unflagged; existing evidential deep learning uncertainty is counter-intuitive.
Approach
Treat classifier outputs as fuzzy membership degrees and define category credibility as the average of possibility and necessity; a label-guided fuzzy multimodal loss avoids over-optimizing necessity for unmatched categories, and per-modality normalized entropy of credibility is merged into a cross-modal uncertainty used to filter retrieval results.
Key results
- FUME achieves mAP@all average of 0.723 on Pascal Sentence and 0.651 on XMediaNet, outperforming all 13 compared methods.
- With uncertainty threshold 0.5 (FUME u=0.5), retrieval mAP rises by 16.9% on Pascal Sentence, 21.3% on Wikipedia, 18.0% on NUS-WIDE-10K, 32.7% on INRIA-Websearch and 21.3% on XMediaNet.
- Ablation on XMediaNet: removing the fuzzy multimodal loss drops average mAP from 0.651 to 0.365 (consistency loss alone), showing Lfml contributes most.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FUME instead of RONO | Cross-modal retrieval · Pascal Sentence | mAP@all (Avg.): 0.706 -> 0.723 | Helped |
| Filter retrieved results with cross-modal uncertainty above 0.5 | FUME · Pascal Sentence | mAP@all (Avg.): 0.723 -> 0.892 | Helped |
| Use FUME instead of MARS | Cross-modal retrieval · Wikipedia | mAP@all (Avg.): 0.522 -> 0.530 | Helped |
| Filter retrieved results with cross-modal uncertainty above 0.5 | FUME · Wikipedia | mAP@all (Avg.): 0.530 -> 0.743 | Helped |
| Use FUME instead of GNN4CMR | Cross-modal retrieval · NUS-WIDE-10K | mAP@all (Avg.): 0.593 -> 0.598 | Helped |
| Filter retrieved results with cross-modal uncertainty above 0.5 | FUME · NUS-WIDE-10K | mAP@all (Avg.): 0.598 -> 0.778 | Helped |
| Use FUME instead of SDML | Cross-modal retrieval · INRIA-Websearch | mAP@all (Avg.): 0.557 -> 0.577 | Helped |
| Filter retrieved results with cross-modal uncertainty above 0.5 | FUME · INRIA-Websearch | mAP@all (Avg.): 0.577 -> 0.904 | Helped |
| Use FUME instead of MARS | Cross-modal retrieval · XMediaNet | mAP@all (Avg.): 0.579 -> 0.651 | Helped |
| Filter retrieved results with cross-modal uncertainty above 0.5 | FUME · XMediaNet | mAP@all (Avg.): 0.651 -> 0.864 | Helped |
| Add consistency learning loss Lcl (ablation) | FUME · NUS-WIDE-10K | mAP@all (Avg.): 0.586 -> 0.598 | Helped |
| Add consistency learning loss Lcl (ablation) | FUME · XMediaNet | mAP@all (Avg.): 0.645 -> 0.651 | Helped |
Surprising findings
- EDL uncertainty is essentially unrelated to prediction entropy and clusters in 0.93-1.00 on XMediaNet despite 0.768 accuracy, suggesting overestimation on many-category datasets.
Limitations
- Uncertainty estimation is evaluated only on image and text modalities and five datasets.
- Uses fixed pre-extracted VGG and word2vec features rather than end-to-end modern encoders.
- Naive direct optimization of credibility is shown to misfit; the label-guided loss relies on a hand-designed piecewise formulation.
Experimental setup
Datasets: Pascal Sentence Wikipedia NUS-WIDE-10K INRIA-Websearch XMediaNet
Benchmarks: Pascal Sentence Wikipedia NUS-WIDE-10K INRIA-Websearch XMediaNet
Metrics: mAP@all deletion rate filtering curves entropy-uncertainty correlation out-of-distribution uncertainty density
Baselines: MCCA ACMR DSCMR SDML MAN DRSL ALGCN ELRCMR MARS GNN4CMR RONO SCL
Builds on: Evidential Deep Learning DSCMR Fuzzy Set Theory contrastive learning
Closest papers by meaning
- BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.86)
- Deep Evidential Hashing for Trustworthy Cross-Modal Retrieval (AAAI 2025 · similarity 0.86)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.85)
- Deep Fuzzy Multi-view Learning for Reliable Classification (ICML 2025 · similarity 0.84)
- Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities (AAAI 2025 · similarity 0.84)
- Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.