Research map / Video / Video-language understanding
Contextual Augmented Global Contrast for Multimodal Intent Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAGC, which models intra- and cross-video global context via a cross-video bank and context-augmented transformer, plus a global context-guided contrastive loss, for multimodal intent recognition on MIntRec and CMU-MOSI.
Problem
Multimodal intent recognition suffers from intent ambiguity and perception biases because existing methods model each video independently and ignore global context across videos, and align modalities only with local (sample or mini-batch) contrast.
Approach
Retrieve similar-scene videos into a cross-video bank filtered by intent-consistency voting, use a context-augmented transformer to fuse this cross-video context into each video's features, and contrast text against global cross-video fusion features across the whole dataset as supervision.
Key results
- On MIntRec twenty-class test-std, CAGC reaches 73.39% ACC vs. 72.29% for MISA, and 90.11% binary ACC vs. 89.19% for MulT.
- On CMU-MOSI, CAGC reports 85.70% ACC-2 and 85.60% F1, slightly above ConFEDE (85.52%, 85.52%).
- Ablation: removing the CAT module drops twenty-class ACC from 73.39% to 71.31%; removing global contrast Lg drops it to 72.22%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CAGC instead of the strongest listed baseline | MIntRec · MIntRec test-std, twenty-class | ACC: 72.65 -> 73.39 | Helped |
| use CAGC instead of the strongest listed baseline | MIntRec · MIntRec test-std, twenty-class | F1: 69.32 -> 70.09 | Helped |
| use CAGC instead of the strongest listed baseline | MIntRec · MIntRec test-std, binary | ACC: 89.24 -> 90.11 | Helped |
| use CAGC instead of the strongest listed baseline | MIntRec · MIntRec test-std, binary | F1: 89.10 -> 90.01 | Helped |
| use CAGC instead of the strongest listed baseline | CMU-MOSI · CMU-MOSI test-std | ACC-2: 85.52 -> 85.70 | Helped |
| use CAGC instead of the strongest listed baseline | CMU-MOSI · CMU-MOSI test-std | F1-Score: 85.52 -> 85.60 | Helped |
| use CAGC instead of the strongest listed baseline | CMU-MOSI · CMU-MOSI test-std | MAE: 0.708 -> 0.775 | Hurt |
| use CAGC instead of the strongest listed baseline | CMU-MOSI · CMU-MOSI test-std | Corr: 0.796 -> 0.774 | Hurt |
| use CAGC instead of the strongest listed baseline | CMU-MOSI · CMU-MOSI test-std | ACC-7: 46.67 -> 44.80 | Hurt |
| add CAT (ablation) | CAGC · MIntRec test-std, twenty-class | ACC: 71.31 -> 73.39 | Helped |
| add CAT (ablation) | CAGC · MIntRec test-std, binary | ACC: 87.84 -> 90.11 | Helped |
| add global contrast loss Lg (ablation) | CAGC · MIntRec test-std, twenty-class | ACC: 72.22 -> 73.39 | Helped |
Surprising findings
- On CMU-MOSI, CAGC has a higher MAE (0.775) than several baselines such as HyCon (0.713) and Self-MM (0.708), so gains are not uniform across metrics.
Limitations
- On CMU-MOSI, MAE and ACC-7 are not best among compared methods, and gains over ConFEDE are small (about 0.2 points).
- Evaluated only on two benchmarks, with MIntRec being small (2,224 samples).
Experimental setup
Datasets: MIntRec CMU-MOSI
Benchmarks: MIntRec CMU-MOSI
Metrics: ACC F1 Precision Recall MAE Corr ACC-2 ACC-7
Baselines: MAG-BERT MulT MISA TFN MFN ICCN HyCon Self-MM ConFEDE
Builds on: MISA MulT MAG-BERT InfoNCE ConFEDE
Closest papers by meaning
- Adaptive Multimodal Fusion: Dynamic Attention Allocation for Intent Recognition (AAAI 2025 · similarity 0.88)
- Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition (AAAI 2024 · similarity 0.88)
- MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations (ICLR 2024 · similarity 0.87)
- Uncertain Multimodal Intention and Emotion Understanding in the Wild (CVPR 2025 · similarity 0.85)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.85)
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.