Research map / Vision-language and multimodal / Cross-modal learning
SHE: Streaming-media Hashing Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHE, a cross-modal hashing paradigm for streaming-media data that trains each modality's sub-network in parallel by mining, transferring, and aligning class-prototype knowledge libraries.
Problem
Cross-modal hashing assumes all modalities are available simultaneously; when modalities arrive chronologically (streaming-media data), existing CMH methods must retrain on all modalities, limiting scalability and flexibility.
Approach
Each modality gets a learnable prototype knowledge library (multiple binary prototypes per class) mined independently; new modalities align their prototypes to the first modality's library with a similarity boundary, plus a discriminative hashing loss, so only the new modality's sub-network is trained.
Key results
- SHE reaches mean MAP 58.5 on Wikipedia and 75.5 on NUS-WIDE (I2T/T2I averages at 128 bits per Table 2 averages), versus 55.0 for the best prior hashing baseline SCH on Wikipedia.
- On XMedia at 128 bits, SHE achieves average MAP 75.0 versus 72.0 for MARS, the strongest real-valued baseline.
- On XMediaNet at 128 bits, SHE achieves average MAP 51.0 versus 46.1 for MARS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add KLM loss (ablation) | SHE · Wikipedia, NUS-WIDE, XMedia, and XMediaNet | MAP: 21.6 -> 68.6; 12.2 -> 69.5; 13.4 -> 75.0; 0.9 -> 51.0 | Helped |
| add KLR loss (ablation) | SHE · Wikipedia, NUS-WIDE, XMedia, and XMediaNet | MAP: 54.3 -> 68.6; 68.2 -> 69.5; 73.8 -> 75.0; 33.3 -> 51.0 | Helped |
| add KLT loss (ablation) | SHE · Wikipedia, NUS-WIDE, XMedia, and XMediaNet | MAP: 68.0 -> 68.6; 69.0 -> 69.5; 39.8 -> 75.0; 29.0 -> 51.0 | Helped |
| add DHL loss (ablation) | SHE · Wikipedia, NUS-WIDE, XMedia, and XMediaNet | MAP: 56.7 -> 68.6; 64.3 -> 69.5; 38.2 -> 75.0; 31.4 -> 51.0 | Helped |
| use multiple prototypes per category rather than one (K=3 versus K=1) (ablation) | SHE · XMedia | MAP | Helped |
| use a similarity boundary rather than forcing identical prototypes (σ=0.95 versus σ=1.0) (ablation) | SHE · XMedia | MAP | Helped |
| choose Audio as the first modality for knowledge-library mining instead of Image (ablation) | SHE · XMedia | MAP: 75.0 -> 24.9 | Hurt |
Surprising findings
- Paired-oriented hashing methods degrade substantially on unpaired multi-modal datasets (XMedia, XMediaNet), often performing far below the real-valued MARS baseline.
- Choosing audio (29-d MFCC) as the first modality on XMedia causes a marked performance drop (mean MAP 24.9 vs about 74 for other first modalities).
Limitations
- Does not account for streaming data with noisy annotations.
- Performance drops when the first modality has low-dimensional features (e.g., 29-d MFCC audio).
Experimental setup
Datasets: Wikipedia NUS-WIDE XMedia XMediaNet
Benchmarks: Wikipedia NUS-WIDE XMedia XMediaNet
Metrics: MAP precision-recall curves
Baselines: MARS GNN4CMR HOPE DGCPN CIRH UCCH HMAH CMMQ MIAN DSPH DHRL DHaPH
Builds on: MARS CMMQ SCH
Closest papers by meaning
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.88)
- Semi-Supervised Online Cross-Modal Hashing (AAAI 2025 · similarity 0.88)
- Dual Self-Paced Cross-Modal Hashing (AAAI 2024 · similarity 0.88)
- Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels (AAAI 2025 · similarity 0.88)
- Deep Evidential Hashing for Trustworthy Cross-Modal Retrieval (AAAI 2025 · similarity 0.87)
- Asymmetric Cross-Modal Hashing Based on Formal Concept Analysis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.