Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
SIG: Speaker Identification in Literature via Prompt-Based Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIG, a BART-based prompt-driven generation method for open-world speaker identification in literary quotations, scoring candidate speakers by generation probability and adding addressee prediction as an auxiliary task.
Problem
Speaker identification for quotations in literary narratives, especially cross-domain (unseen speakers/novels) and non-explicit cases where the speaker is not mentioned in surrounding context.
Approach
Verbalize speaker identification as a prompt-based seq2seq generation task with a mask placeholder after the quotation, and classify by the average generation probability of each candidate speaker filled into the target template, which allows any candidate form and out-of-domain speakers; addressee prediction is added as an auxiliary prompt task.
Key results
- On PDNC cross-domain evaluation, SIG reaches 0.72 mean accuracy overall and 0.70 on non-explicit quotations, versus 0.71 for zero-shot ChatGPT and 0.68 for BookNLP.
- Addressee prediction as an auxiliary task raises PDNC accuracy from 68.43 to 71.59 with BART.
- On WP in-domain evaluation, SIG achieves 86.15 test accuracy, versus 82.50 for CSN and 80.90 for E2E-SI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use classification by generation instead of direct generation | SIG on PDNC cross-domain non-explicit quotations · PDNC cross-domain, non-explicit | accuracy: 0.56/0.51 -> 0.70/0.73 | Helped |
| use classification by generation instead of direct generation | SIG on PDNC cross-domain total quotations · PDNC cross-domain, total | accuracy: 0.57/0.54 -> 0.72/0.78 | Helped |
| use SIG instead of ChatGPT | PDNC cross-domain total quotations · PDNC cross-domain, total | accuracy: 0.71/0.71 -> 0.72/0.78 | Helped |
| use SIG instead of RoBERTa | PDNC in-domain, Top 1 · PDNC in-domain, Top 1 | accuracy: 69.39 -> 64.45 | Hurt |
| use SIG instead of RoBERTa | PDNC in-domain, Top 5 · PDNC in-domain, Top 5 | accuracy: 84.55 -> 86.31 | Helped |
| use SIG instead of E2E-SI | WP test · WP test | accuracy: 80.90 -> 86.15 | Helped |
| add source and target prompt templates rather than using none (ablation) | SIG on PDNC · PDNC | accuracy: 56.34 -> 68.43 | Helped |
| add a source prompt while leaving the target untemplated (ablation) | SIG on PDNC · PDNC | accuracy: 56.34 -> 58.12 | Helped |
| use replied by in both source and target templates instead of the best template pair (ablation) | SIG on PDNC · PDNC | accuracy: 68.43 -> 66.32 | Hurt |
| use Speaker in both source and target templates instead of the best template pair (ablation) | SIG on PDNC · PDNC | accuracy: 68.43 -> 64.44 | Hurt |
| add fiction identification auxiliary task (ablation) | SIG on PDNC · PDNC | accuracy: 68.43 -> 63.38 | Hurt |
| add gender identification auxiliary task (ablation) | SIG on PDNC · PDNC | accuracy: 68.43 -> 69.51 | Helped |
Surprising findings
- Zero-shot ChatGPT is a strong baseline that outperforms BookNLP and BookNLP+ on PDNC.
- In-domain, the RoBERTa classifier beats SIG at top-1 accuracy (69.39 vs 64.45), with SIG winning only at top-4 and top-5.
Limitations
- Does not use open-source LLMs for training due to computational constraints.
- Direct generation performs much worse than classification by generation on PDNC (0.57 vs 0.72).
- ChatGPT results are single-run and use a lenient substring-match metric, so they are an upper bound.
Experimental setup
Datasets: PDNC WP
Benchmarks: PDNC WP
Metrics: accuracy top-k accuracy
Baselines: BookNLP BookNLP+ ChatGPT RoBERTa CSN E2E-SI
Builds on: BART BookNLP RoBERTa chain-of-thought
Closest papers by meaning
- Advancing Audio-Based Text Generation with Imbalance Preference Optimization (AAAI 2025 · similarity 0.82)
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.82)
- Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning (AAAI 2024 · similarity 0.81)
- Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations (AAAI 2024 · similarity 0.81)
- Agnostic Language Identification and Generation (NeurIPS 2026 · similarity 0.81)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.