Research map / Vision-language and multimodal / Cross-modal learning

Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DUMoE, a drift-aware multimodal user representation model combining a multi-scale temporal backbone with a sparse mixture-of-experts interest adapter and three-stage training, evaluated on a new X (Twitter) dataset for interest classification and interaction prediction.

Problem

Learning user representations from noisy, multimodal social media histories where user interests drift across multiple temporal scales and multiple interests co-exist.

Approach

Combines a backbone fusing static profile, short-term LSTM and long-term Transformer features with a sparse top-K' MoE of residual adapter experts gated per user, trained in three stages (backbone, expert specialization on dominant-interest users, gating only).

Key results

• On interest classification, DUMoE reaches Hit@1 0.872 and Recall 0.856, versus 0.821 Hit@1 for the strongest baseline MIND.
• On interaction prediction, DUMoE reaches Recall 0.926 and F1 0.888, above PeterRec (Recall 0.894, F1 0.885); accuracy 0.883 is comparable to PTUM at 0.886.
• Ablations show removing the MoE adapter drops Hit@1 to 0.858 and removing stage-wise training lowers KL-based performance (KL 0.341 vs 0.323).

Surprising findings

• Long-term-only backbone variant nearly matches the full model on interest classification (Hit@1 0.862 vs 0.872).
• On interaction prediction, DUMoE's accuracy (0.883) is not the best; PTUM (0.886) is higher.

Limitations

• Pseudo-labels generated by GPT-5.1 may carry annotation noise and model bias; only manual spot-check verification described.
• Fixed short-term and long-term window lengths and a fixed number of experts with static top-K' routing.
• Offline training regime; no online continual learning evaluated.

Experimental setup

Datasets: DUMoE X dataset (collected from X/Twitter)

Benchmarks: DUMoE X dataset

Metrics: Hit@1 NDCG@3 Recall KL divergence Accuracy F1 BCE

Baselines: SASRec BERT4Rec PTUM MIND HORAE PeterRec UniSRec

Builds on: UNITE UniSRec PTUM HORAE MIND

Closest papers by meaning

• Improving Multimodal Social Media Popularity Prediction via Selective Retrieval Knowledge Augmentation (AAAI 2025 · similarity 0.83)
• Unity by Diversity: Improved Representation Learning for Multimodal VAEs (NeurIPS 2024 · similarity 0.83)
• Curriculum Conditioned Diffusion for Multimodal Recommendation (AAAI 2025 · similarity 0.83)
• Density-based User Representation using Gaussian Process Regression for Multi-interest Personalized Retrieval (NeurIPS 2024 · similarity 0.83)
• THGFormer: Time-Aware Hypergraph Learning for Multimodal Social Media Popularity Prediction (Student Abstract) (AAAI 2024 · similarity 0.82)
• MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation (AAAI 2025 · similarity 0.82)