Research map / Large language models / AI and society
Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ToM-in-AMC, a few-shot character-guessing benchmark built from ~1,000 movie scripts to test meta-learning of theory-of-mind, and proposes ToMPro, an LLM prompting method that generates multi-dimensional mental states before guessing speakers.
Problem
Existing theory-of-mind assessments for machines use synthetic settings or large per-character data, ignoring the few-shot meta-learning nature of how humans understand new fictional characters.
Approach
Treat each movie as a few-shot task where models identify anonymized speakers after seeing a few starting scenes, and propose ToMPro, which iteratively generates personality, emotion, belief, desire and intention descriptions with GPT-4 and then guesses identities from them.
Key results
- GPT-4 ToMPro achieves 68.2% dev accuracy and 66.9% test accuracy, versus 59.4% dev / 58.6% test for LEOPARD and 55.4% dev / 53.2% test for the prototypical network.
- Human raters reach 88.0% accuracy on a development subset, more than 20 points above the best model.
- Removing updates after training scenes drops ToMPro from 68.2% to 61.8%; replacing GPT-4 with GPT-3.5 drops it to 60.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 ToMPro instead of GPT-4 ICL (20-shot) | GPT-4 character understanding · ToM-in-AMC Dev, human-study subset | accuracy: 67.8 -> 68.2 | Helped |
| Train the Prototypical Network on ToM-in-AMC rather than TVSG (ablation) | Prototypical Network · ToM-in-AMC Test | accuracy: 46.4 -> 53.2 | Helped |
| Increase GPT-4 ICL demonstrations from 10-shot to 20-shot (ablation) | GPT-4 ICL · ToM-in-AMC Dev, human-study subset | accuracy: 63.8 -> 67.8 | Helped |
| Replace GPT-3.5 with GPT-4 for 10-shot ICL (ablation) | LLM in-context learning · ToM-in-AMC Dev, human-study subset | accuracy: 54.9 -> 63.8 | Helped |
| Add ToMPro's updates after training scenes (ablation) | GPT-4 ToMPro · ToM-in-AMC Dev, human-study subset | accuracy: 61.8 -> 68.2 | Helped |
| Replace GPT-3.5 with GPT-4 in ToMPro (ablation) | ToMPro · ToM-in-AMC Dev, human-study subset | accuracy: 60.3 -> 68.2 | Helped |
| Include multiple ToM dimensions in character descriptions (ablation) | ToMPro · ToM-in-AMC | accuracy | Helped |
| Emphasize desire and intention dimensions (ablation) | ToMPro · ToM-in-AMC | accuracy | Helped |
| Emphasize emotion dimension (ablation) | ToMPro · ToM-in-AMC | accuracy | Helped |
| Apply iterative generation to the four short-term ToM dimensions (ablation) | ToMPro · ToM-in-AMC | accuracy | Hurt |
| Use GPT-4 ToMPro rather than GPT-4 character-memory recollections (ablation) | Inductive character understanding · ToM-in-AMC | accuracy: 67.4 -> 68.2% | Helped |
| Generate ToMPro mental states from non-perturbed scenes (ablation) | GPT-4 ToMPro · ToM-in-AMC | accuracy: 68.2% to 70.7% | Helped |
Surprising findings
- Zero-shot GPT-4 identifying characters from their names alone reaches 69.2%, indicating memorization of the movies despite perturbation.
- GPT-4 ICL on the perturbed setting scores 67.8% dev accuracy, showing strong performance without explicit mental-state modeling.
Limitations
- Relies on GPT-4, which is resource-intensive; GPT-3.5 falls about 8% behind.
- Gender imbalance among main characters lets female characters be identified without historical information.
- Generated desire descriptions are poor due to limited global understanding of characters.
Experimental setup
Datasets: ToM-in-AMC training split (IMSDB movie scripts) TVSG TV show transcripts
Benchmarks: ToM-in-AMC
Metrics: instance-level accuracy
Baselines: Prototypical Network Multi-task learning classifiers LEOPARD GPT-4 in-context learning GPT-3.5 in-context learning GPT-4 character memory
Builds on: Prototypical Networks LEOPARD MAML Longformer
Closest papers by meaning
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.86)
- From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models (ICML 2025 · similarity 0.85)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.85)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.84)
- SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions (NeurIPS 2025 · similarity 0.84)
- Visual Objectification in Films: Towards a New AI Task for Video Interpretation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.