Research map / Vision-language and multimodal / Cross-modal learning
MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIntRec2.0, a large multi-party, multi-turn multimodal (text/video/audio) intent recognition benchmark with 30 intent classes and out-of-scope samples, plus baselines (BERT, MAG-BERT, MulT), ChatGPT and human evaluation.
Problem
Existing multimodal intent datasets are small, single-turn, and lack out-of-scope samples, limiting research on context-aware intent recognition and open-world out-of-scope detection.
Approach
A large-scale dataset of 1,245 multi-party TV-series dialogues with 15,040 utterances (9,304 in-scope, 5,736 OOS) annotated with an expanded 30-class intent taxonomy and speaker identities, plus a unified framework that concatenates same-speaker context and combines outlier exposure loss with DOC-based thresholding for OOS detection.
Key results
- Multimodal fusion (MulT) improves single-turn in-scope F1 from 51.60 (TEXT) to 54.12 and MAG-BERT to 55.17 without OOS training.
- Training with OOS samples raises MulT's F1-OOS to 61.66 from a text-only baseline of 61.54 in the joint setting, while multi-turn context gives little or no gain on joint classification.
- Humans with 100 dialogues reach 71.03% in-scope accuracy versus ChatGPT's 35.27% zero-shot, a gap over 30 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace text-only features with MAG-BERT multimodal fusion | BERT-LARGE intent classification · MIntRec2.0 single-turn, w/o OOS | F1: 51.60 -> 55.17 | Helped |
| Replace text-only features with MAG-BERT multimodal fusion | BERT-LARGE intent classification · MIntRec2.0 single-turn, w OOS | F1: 52.08 -> 53.64 | Helped |
| Add same-speaker context by concatenating previous utterances (ablation) | MIntRec2.0 multimodal intent classification · MIntRec2.0 multi-turn, w OOS | F1: TEXT: 52.72 -> 53.61; MAG-BERT: 53.64 -> 53.89; MulT: 52.72 -> 53.96 | Mixed |
| Add same-speaker context by concatenating previous utterances (ablation) | MIntRec2.0 multimodal intent classification · MIntRec2.0 multi-turn, in-scope + OOS | F1: Context MAG-BERT: 48.00 -> 47.25; Context MulT: 47.35 -> 46.98 | Mixed |
| Include OOS samples during training with outlier exposure (ablation) | MIntRec2.0 intent classification and OOS detection · MIntRec2.0 single-turn | F1-OOS: increase of over 30% | Helped |
| Use multimodal fusion instead of text-only features | MIntRec2.0 multi-turn intent classification · MIntRec2.0 multi-turn, in-scope | F1: 53.61 -> 53.89; 53.61 -> 53.96 | Helped |
| Provide 10 dialogues as prior knowledge | ChatGPT · MIntRec2.0 few-shot, in-scope + OOS | F1-OOS: 21.21 -> 27.85 | Helped |
| Provide 10 dialogues as prior knowledge | ChatGPT · MIntRec2.0 few-shot, in-scope | ACC: 35.27 -> 34.53 | Hurt |
| Increase human prior knowledge from 10 to 100 dialogues | Human evaluators · MIntRec2.0 few-shot, in-scope | ACC: 64.34 -> 71.03 | Helped |
| Train with OOS samples as a separate K+1 class instead of outlier exposure (ablation) | MIntRec2.0 intent classification · MIntRec2.0, in-scope | F1: approximately 10% | Hurt |
| Use DialogueRNN to model temporal context with outlier exposure | MIntRec2.0 multi-turn classification · MIntRec2.0 multi-turn, in-scope + OOS | F1: 2.91 | Hurt |
Surprising findings
- Context modeling with fused multimodal representations (DialogueRNN) collapses to trivial solutions, mostly predicting OOS.
- Multimodal fusion gives little or no improvement over text-only once multi-turn context and OOS samples are included.
Limitations
- Single-intent assumption; only 4.2% of utterances had a second intent in a small check.
- Data drawn from only three TV series, limiting domain diversity.
- Multimodal fusion and context modeling yield limited gains in the multi-turn OOS setting.
Experimental setup
Datasets: MIntRec2.0 MIntRec
Benchmarks: MIntRec2.0
Metrics: F1 precision recall accuracy weighted F1 weighted precision F1-IS F1-OOS
Baselines: TEXT (BERT-large) MAG-BERT MulT DialogueRNN ChatGPT MAG-BERT-10
Builds on: MIntRec MAG-BERT MulT DOC Outlier Exposure
Closest papers by meaning
- Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling via Adjustive and Forced Cross-Task Alignment (AAAI 2024 · similarity 0.88)
- Towards Multi-Intent Spoken Language Understanding via Hierarchical Attention and Optimal Transport (AAAI 2024 · similarity 0.88)
- Uni-MIS: United Multiple Intent Spoken Language Understanding via Multi-View Intent-Slot Interaction (AAAI 2024 · similarity 0.87)
- Contextual Augmented Global Contrast for Multimodal Intent Recognition (CVPR 2024 · similarity 0.87)
- Divide-Solve-Combine: An Interpretable and Accurate Prompting Framework for Zero-shot Multi-Intent Detection (AAAI 2025 · similarity 0.87)
- Multi-Granularity Open Intent Classification via Adaptive Granular-Ball Decision Boundary (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.