Research map / Large language models / Language, safety and interpretability
Topic-Conversation Relevance (TCR) Dataset and Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Topic-Conversation Relevance (TCR) dataset of ~1,500 meetings with pre-meeting-style topics, plus GPT-4-based benchmarks for judging whether transcript snippets are on topic.
Problem
Measuring whether a meeting conversation stays relevant to pre-defined agenda topics is unexplored, and existing meeting summarization datasets mostly provide post-meeting minutes rather than pre-meeting agenda topics.
Approach
Builds a multi-source meeting dataset with topic annotations converted to pre-meeting agenda style (GPT-4 rewriting of MeetingBank summaries), a flexible JSON schema with topic add/remove variations, and GPT-4 benchmarks framing relevance as binary "discussed" vs "not discussed" classification.
Key results
- GPT-4 benchmark F1 reaches 0.98 on MeetingBank_rnd30 (5-minute snippets) and 0.96 on SIM_syn100 (5-minute snippets) for the Not Discussed class.
- Performance is lower on less structured meetings: ICSI_QMSum F1 0.75 at 5 minutes, dropping to 0.72 at 10 minutes.
- Re-annotated MeetingBank topics received an average human MOS of 4.6 out of 5 on 100 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase transcript snippet length from 5 to 15 minutes | GPT-4 topic-relevance benchmark · SIM_syn100, ICSI_original75, ICSI_QMSum, MeetingBank_rnd30, MeetingBank_ReAnnotated_rnd30, NCPC, and ELITR | F1: SIM_syn100: 0.9587 -> 0.9175; ICSI_original75: 0.8663 -> 0.8488; ICSI_QMSum: 0.7506 -> 0.7222; MeetingBank_rnd30: 0.9804 | Hurt |
| Use re-annotated pre-meeting topics instead of original meeting-minute topics | GPT-4 on MeetingBank_rnd30 · MeetingBank_rnd30, all snippet lengths | F1: 0.9804 -> 0.9817 (5 min); 0.9767 -> 0.9810 (10 min); 0.9688 -> 0.9755 (15 min) | Helped |
| Evaluate snippets containing multiple topics instead of only one topic | GPT-4 topic-relevance benchmark · MeetingBank_rnd30 and other reported datasets | F1: MeetingBank_rnd30, 5 min: 0.9845 -> 0.9663 | Hurt |
Surprising findings
- Multi-topic snippets are harder than single-topic snippets for GPT-4 on less structured meetings, attributed to unclear topic transitions.
- GPT-4 was reported as unreliable for meeting summary evaluation in the AutoMin challenge, per the organizers' findings cited in related work.
Limitations
- Topic boundaries in transcripts are blurred, so chunk-level labels may be noisy.
- Coverage is limited to English and to a few meeting domains; real workplace meetings are underrepresented due to sensitivity.
- Benchmarks rely on a single LLM (GPT-4) with temperature 0 and no error bars.
Experimental setup
Datasets: SIM ICSI QMSum MeetingBank NCPC ELITR
Benchmarks: TCR benchmark (SIM_syn100, ICSI, QMSum_ICSI, MeetingBank_rnd30, MeetingBank_ReAnnotated_rnd30, NCPC, ELITR)
Metrics: F1 precision recall
Builds on: QMSum MeetingBank ICSI Corpus AMI corpus ELITR
Closest papers by meaning
- Are Large Language Models Ready for Multi-Turn Tabular Data Analysis? (ICML 2025 · similarity 0.81)
- Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.80)
- cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers (NeurIPS 2024 · similarity 0.80)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.80)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.79)
- ChatQA: Surpassing GPT-4 on Conversational QA and RAG (NeurIPS 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.