Research map / Vision-language and multimodal / Cross-modal learning

A New Formula for Sticker Retrieval: Reply with Stickers in Multi-Modal and Multi-Session Conversation

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MultiChat, a multi-modal multi-session Chinese sticker-retrieval dataset, and IGSR, an intent-guided framework that builds an LLM memory bank from sticker-attributed history and retrieves stickers with a CLIP-based multi-task model.

Problem

Sticker retrieval for replies has been studied only over single sessions or previous text, ignoring multi-modal (sticker) and multi-session conversation history.

Approach

Augment utterances with per-sticker attributes (intent, style, gesture, posture, expression, verbal) to build an LLM-generated memory bank of past sessions; retrieve relevant memories for the current session, predict user intent from memory and context representations, and jointly train intent classification with CLIP-based contrastive sticker retrieval.

Key results

• IGSR achieves P@1 14.45%, P@5 54.94%, mAP 19.39% on MultiChat, beating the best baseline on P@1 (ChatGLM3 14.09%... Qwen-VL 9.25/LLaVA 10.23 for multimodal) and the strongest P@5 baseline (Qwen-VL 48.38%).
• Removing memory drops P@1 from 14.45 to 7.36 and GPT-4 score from 64.04 to 29.38; removing intent drops P@1 to 2.69.
• Human evaluation score for IGSR is 67.54% versus 64.04% for the strongest multimodal baseline (PCME/CLIP at 64.04%) on a 114-sample subset.

Measured effects reported in the paper

Surprising findings

• Removing relevant memory improves Baichuan2, Qwen1.5, ChatGLM3 and Qwen-VL, attributed to overly long inputs.
• GPT-4 and human evaluations disagree on the ranking of text-based versus multi-modal methods.

Limitations

• Dataset is Chinese WeChat group chats from only five groups, limiting generality.
• Does not model user identity or individual user preferences, which the authors note hurts some cases.
• Evaluation relies on P@N and mAP with intent-label matching and a small 114-sample human/GPT-4 subset.

Experimental setup

Datasets: MultiChat

Benchmarks: MultiChat

Metrics: P@1 P@3 P@5 mAP GPT-4 score human evaluation

Baselines: Baichuan2 Qwen1.5 Llama3 ChatGLM3 MOD SRS IRRA LGUR PCME CLIP Qwen-VL GPT-4

Builds on: CLIP InfoNCE Qwen-VL

Closest papers by meaning

• SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents (ICLR 2025 · similarity 0.84)
• MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations (ICLR 2024 · similarity 0.84)
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025 · similarity 0.84)
• MemoryBank: Enhancing Large Language Models with Long-Term Memory (AAAI 2024 · similarity 0.82)
• Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment (CVPR 2025 · similarity 0.82)
• Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding (AAAI 2025 · similarity 0.82)