Research map / Vision-language and multimodal / Human and embodied modeling
Improving Multimodal Social Media Popularity Prediction via Selective Retrieval Knowledge Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SKAPP, a selective retrieval-augmented framework for multimodal social media popularity prediction that uses a metadata-aware meta retriever, an RRCP-based selective refiner, and vision-language GNNs with RRCP-attention fusion.
Problem
Predicting popularity of multimodal user-generated content (UGC) on social media, where existing methods treat each UGC in isolation and simple semantic retrieval introduces irrelevant or noisy context.
Approach
A meta retriever builds BM25 queries from vision-language descriptions plus metadata (user, tags, posting info), and a selective refiner keeps only retrieved UGCs whose Relative Retrieval Contribution to Prediction (RRCP) is non-negative; the kept UGCs are aggregated by vision and language GNNs and fused with RRCP-weighted attention.
Key results
- SKAPP reaches MSE 0.9662 on ICIP, 1.8196 on SMPD and 2.0936 on Instagram, versus second-best MSE of 1.5614 (w/o retrieval baseline ablation) / 1.7600 (MMRA) on ICIP.
- Reported relative MSE improvements of 39.61% (ICIP), 48.19% (SMPD), and 46.94% (Instagram) over the second-best method.
- Ablation shows removing the meta retriever raises ICIP MSE from 0.9662 to 1.9006, and removing the selective refiner raises it to 1.1004.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SKAPP selective retrieval knowledge augmentation | multimodal popularity prediction · ICIP | MSE: 1.7600 -> 0.9662 | Helped |
| Add SKAPP selective retrieval knowledge augmentation | multimodal popularity prediction · SMPD | MSE: 3.5119 -> 1.8196 | Helped |
| Add SKAPP selective retrieval knowledge augmentation | multimodal popularity prediction · Instagram | MSE: 3.9456 -> 2.0936 | Helped |
| Add retrieval (ablation) | SKAPP · ICIP | MSE: 1.5614 -> 0.9662 | Helped |
| Add retrieval (ablation) | SKAPP · SMPD | MSE: 4.0443 -> 1.8196 | Helped |
| Add retrieval (ablation) | SKAPP · Instagram | MSE: 3.2734 -> 2.0936 | Helped |
| Add the meta retriever (ablation) | SKAPP · ICIP | MSE: 1.9006 -> 0.9662 | Helped |
| Add the meta retriever (ablation) | SKAPP · SMPD | MSE: 4.1353 -> 1.8196 | Helped |
| Add the meta retriever (ablation) | SKAPP · Instagram | MSE: 5.2537 -> 2.0936 | Helped |
| Add the selective refiner (ablation) | SKAPP · ICIP | MSE: 1.1004 -> 0.9662 | Helped |
| Add the selective refiner (ablation) | SKAPP · SMPD | MSE: 2.0854 -> 1.8196 | Helped |
| Add the selective refiner (ablation) | SKAPP · Instagram | MSE: 2.6332 -> 2.0936 | Helped |
Surprising findings
- Semantic-only retrieval based on photos or texts performs worse than metadata-based retrieval in the ablation.
- Retrieval adds substantial computational cost: about 7 hours of retrieval for a 300K-UGC dataset, though the authors argue the gain justifies it.
Limitations
- Retrieval is computationally expensive (hours per dataset) compared to lightweight baselines.
- Performance degrades with reduced training data and polluted UGCs/retrievals, though SKAPP remains better than baselines.
- Selective refinement uses directional contribution estimates that rely on a trained model, rather than a specialized estimator.
Experimental setup
Datasets: ICIP SMPD Instagram
Benchmarks: ICIP SMPD Instagram
Metrics: MSE MAE Spearman's rank correlation
Baselines: SVR HyFea MFTM CLSTM HMMVED DLBA MASSL BLIP CBAN NIPA MMRA
Builds on: GraphAdapter BM25 conditional cross-mutual information NIPA MMRA
Closest papers by meaning
- THGFormer: Time-Aware Hypergraph Learning for Multimodal Social Media Popularity Prediction (Student Abstract) (AAAI 2024 · similarity 0.90)
- SMTPD: A New Benchmark for Temporal Prediction of Social Media Popularity (CVPR 2025 · similarity 0.88)
- In-context Prompt-augmented Micro-video Popularity Prediction (AAAI 2025 · similarity 0.86)
- LGMRec: Local and Global Graph Learning for Multimodal Recommendation (AAAI 2024 · similarity 0.84)
- DOGE: LLMs-Enhanced Hyper-Knowledge Graph Recommender for Multimodal Recommendation (AAAI 2025 · similarity 0.84)
- CasFT: Future Trend Modeling for Information Popularity Prediction with Dynamic Cues-Driven Diffusion Models (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.