Research map / Vision-language and multimodal / Human and embodied modeling
Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Friends-MMC, a multi-modal multi-party conversation dataset from the TV series Friends with speaker and face annotations, and benchmarks speaker identification (with a solver-based baseline) and speaker-aware response prediction.
Problem
Existing multi-modal dialogue datasets are two-party question-answer style with bystander speakers outside the visual context, and speaker-aware multi-party conversation understanding grounded in video is understudied.
Approach
Build a situated multi-party dataset from TV series with per-turn speakers and face boxes/names, and combine a visual active-speaker model with a text same-speaker model via a quadratic binary optimization solver for speaker identification; show speaker information helps response prediction.
Key results
- Best non-oracle speaker identification is TalkNet + text model with quadratic binary optimization at 83.21% (5 turns) and 83.60% (8 turns) on test, versus 80.89% for TalkNet alone.
- Multi-modal pre-trained and proprietary baselines perform poorly, e.g. Violet 32.66% and GPT-4o 66.36% on 5-turn test; human performance reaches 82.25% (5 turns) and 84.49% (8 turns).
- On response prediction with Llama2-7B, ground-truth speakers raise 5-turn accuracy from 30.69% (no speaker) to 36.89%, and shuffled speakers reach 35.20%, suggesting speaker info mainly serves as local context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DeBERTa-v3 speaker-relation predictions to CNN visual predictions | Speaker identification system with CNN · Friends-MMC test and test-noisy, 5 and 8 turns | accuracy: 72.88 -> 75.81; 63.72 -> 68.61; 72.90 -> 74.53; 62.51 -> 67.21 | Helped |
| Add DeBERTa-v3 speaker-relation predictions to TalkNet visual predictions | Speaker identification system with TalkNet · Friends-MMC test and test-noisy, 5 and 8 turns | accuracy: 80.89 -> 83.21; 70.91 -> 74.12; 81.00 -> 83.60; 70.50 -> 75.00 | Helped |
| Replace learned speaker-relation predictions with ground-truth relations (ablation) | CNN + DeBERTa-v3 speaker identification system · Friends-MMC test and test-noisy, 5 and 8 turns | accuracy: 75.81 -> 84.90; 68.61 -> 78.01; 74.53 -> 90.80; 67.21 -> 83.93 | Helped |
| Replace learned speaker-relation predictions with ground-truth relations (ablation) | TalkNet + DeBERTa-v3 speaker identification system · Friends-MMC test and test-noisy, 5 and 8 turns | accuracy: 83.21 -> 90.88; 74.12 -> 83.09; 83.60 -> 95.10; 75.00 -> 89.69 | Helped |
| Fine-tune TalkNet on Friends-MMC instead of using it zero-shot (ablation) | TalkNet visual model · Friends-MMC speaker identification | accuracy | No clear effect |
| Add ground-truth speaker names (ablation) | Llama2-7B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 30.69 -> 36.89; 36.98 -> 49.36 | Helped |
| Add ground-truth speaker names (ablation) | Emu-14B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 30.49 -> 34.06; 31.09 -> 36.30 | Helped |
| Use predicted TalkNet + DeBERTa-v3 speaker names instead of no speaker information (ablation) | Llama2-7B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 30.69 -> 34.56; 36.98 -> 46.64 | Helped |
| Use predicted TalkNet + DeBERTa-v3 speaker names instead of no speaker information (ablation) | Emu-14B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 30.49 -> 32.97; 31.09 -> 34.64 | Helped |
| Replace ground-truth speaker names with randomly assigned names (ablation) | Llama2-7B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 36.89 -> 31.23; 49.36 -> 43.32 | Hurt |
| Replace ground-truth speaker names with randomly assigned names (ablation) | Emu-14B response prediction · Friends-MMC, 5 and 8 turns | accuracy: 34.06 -> 29.35; 36.30 -> 31.55 | Hurt |
| Shuffle speaker names consistently in speaker information and utterance text (ablation) | Llama2-7B and Emu-14B response prediction · Friends-MMC, 5 and 8 turns | accuracy: Llama2-7B: 35.20, 48.60; Emu-14B: 33.02, 35.17 | Mixed |
Surprising findings
- Fine-tuned multi-modal pre-trained models (Violet, LLaVA, Emu) and GPT-4o underperform a simple vision-plus-text baseline on speaker identification.
- Speaker information mainly helps as a local context indicator rather than as a global speaking-style indicator, based on shuffled-name experiments.
Limitations
- Dataset is drawn from a single TV series (Friends), limiting diversity beyond movies/sitcoms.
- Training face labels are automatically generated with about 95% accuracy on the verification check, introducing label noise; only the test set is manually labelled.
Experimental setup
Datasets: Friends-MMC C1C Ubuntu Dialogue Corpus VGGFace2
Benchmarks: Friends-MMC test Friends-MMC test-noisy
Metrics: accuracy response selection accuracy (1 of 10 candidates)
Baselines: CNN frame-based speaker model TalkNet GPT 3.5 (3-shot) GPT-4o (0-shot) Violet LLaVA v1.5-13B Emu-14B Llama2-7B
Builds on: TalkNet DeBERTa-v3 Facenet-512 MPC-BERT Emu
Closest papers by meaning
- MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation (NeurIPS 2026 · similarity 0.86)
- Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations (CVPR 2024 · similarity 0.86)
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.85)
- MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations (ICLR 2024 · similarity 0.85)
- Multi-speaker Attention Alignment for Multimodal Social Interaction (CVPR 2026 · similarity 0.85)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.