Research map / Vision-language and multimodal / Cross-modal learning
Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCL-MAP, which builds a modality-aware prompt from text, video and audio features and applies token-level NT-Xent contrastive learning between a [MASK] token and the ground-truth label token to improve multimodal intent recognition on MIntRec and MELD-DA.
Problem
Multimodal intent recognition fails to capture deep intent semantics because existing methods ignore cross-modal correlations and learn weak semantic features from nonverbal video and audio modalities.
Approach
A similarity-based modality alignment plus cross-modality attention generates a learnable, instance-conditioned prompt from video and audio; token-level contrastive learning then pairs the [MASK] token with the ground-truth label token so that textual label semantics guide nonverbal feature learning.
Key results
- On MIntRec, TCL-MAP reaches 73.62% ACC and 73.31% WF1, improving over the best baseline (MAG-BERT 72.65% ACC, MulT 72.31% WF1 range) by 0.97 ACC points.
- On MELD-DA, TCL-MAP reaches 61.75% ACC, a 1.12-point gain over the best baseline (MAG-BERT 60.63%).
- Removing token-level contrastive learning drops MIntRec ACC by about 1.5 points in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TCL-MAP | multimodal intent recognition · MIntRec | ACC (%): 0.97% | Helped |
| add TCL-MAP | multimodal intent recognition · MIntRec | WF1 (%): 0.93% | Helped |
| add TCL-MAP | multimodal intent recognition · MIntRec | R (%): 1.22% | Helped |
| add TCL-MAP | multimodal intent recognition · MELD-DA | ACC (%): 1.12% | Helped |
| add similarity-based modality alignment instead of temporal CTC alignment (ablation) | TCL-MAP · MIntRec | ACC (%): 73.15 -> 73.62 | Helped |
| add similarity-based modality alignment instead of temporal CTC alignment (ablation) | TCL-MAP · MELD-DA | WP (%): 59.58 -> 60.33 | Helped |
| add modality-aware prompting (ablation) | TCL-MAP · MIntRec | ACC (%): 72.67 -> 73.62 | Helped |
| add modality-aware prompting (ablation) | TCL-MAP · MIntRec | WF1 (%): 72.31 -> 73.31 | Helped |
| add modality-aware prompting (ablation) | TCL-MAP · MELD-DA | ACC (%): 60.40 -> 61.75 | Helped |
| add token-level contrastive learning loss (ablation) | TCL-MAP · MIntRec | ACC (%): 72.13 -> 73.62 | Helped |
| add token-level contrastive learning loss (ablation) | TCL-MAP · MIntRec | WF1 (%): 71.80 -> 73.31 | Helped |
| add token-level contrastive learning loss (ablation) | TCL-MAP · MELD-DA | ACC (%): 61.26 -> 61.75 | Helped |
Surprising findings
- The [MASK] prompt performs notably worse than handcrafted prompts, suggesting inappropriate prompts can mislead intent understanding.
- TCL-MAP surpasses human performance on some MIntRec intent classes such as Apologise, Thank and Agree, but underperforms on Taunt and Joke.
Limitations
- Gains over baselines are small (about 1 point), and no variance across seeds is reported for the main table.
- Underperforms on Taunt and Joke classes, attributed to limited data and intent complexity.
- Hyperparameters such as the similarity thresholds alpha are not thoroughly analyzed.
Experimental setup
Datasets: MIntRec MELD-DA
Benchmarks: MIntRec MELD-DA
Metrics: ACC WF1 WP recall
Baselines: MAG-BERT MISA MulT
Builds on: CoCoOp MAG-BERT SimCLR MulT MISA
Closest papers by meaning
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.89)
- Contextual Augmented Global Contrast for Multimodal Intent Recognition (CVPR 2024 · similarity 0.88)
- Deep Correlated Prompting for Visual Recognition with Missing Modalities (NeurIPS 2024 · similarity 0.88)
- Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · similarity 0.87)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.