Research map / Large language models / AI and society
RecToM: A Benchmark for Evaluating Machine Theory of Mind in LLM-based Conversational Recommender Systems
Mengfan Li, Xuanhua Shi, Yang Deng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought prompting (ablation) | DeepSeek-V3 · RecToM recommender coarse intention | accuracy: 44.26 -> 46.21 | Improved |
| add chain-of-thought prompting (ablation) | GPT-4o · RecToM seeker coarse intention | accuracy: 64.22 -> 54.10 | Hurt |
| add chain-of-thought prompting (ablation) | GPT-4o · RecToM recommender belief | accuracy: 74.74 -> 75.20 | No clear effect |
| add chain-of-thought prompting (ablation) | DeepSeek-V3 · RecToM recommender belief | accuracy: 69.86 -> 79.46 | Improved |
| add chain-of-thought prompting (ablation) | Claude 3.5 Haiku · RecToM seeker desire | accuracy: 90.68 -> 78.73 | Hurt |
| add chain-of-thought prompting (ablation) | DeepSeek-V3 · RecToM seeker desire | accuracy: 86.05 -> 76.10 | Hurt |
| add chain-of-thought prompting (ablation) | Claude 3.5 Haiku · RecToM recommender judgement recall for No | recall (%): 2.36 -> 10.13 | Improved |
| use multiple-choice instead of single-choice questions | LLMs (model average) · RecToM | accuracy: 27.74 (recommender fine intention) vs 68.72 (belief) | Hurt |
| use fine-grained instead of coarse-grained intention labels | GPT-4o · RecToM seeker intention | accuracy: 64.22 -> 28.84 | Hurt |
| map fine-grained outputs to coarse categories (Fine2Coarse) (ablation) | DeepSeek-V3 · RecToM recommender intention | accuracy: 29.71 -> 45.40 | Improved |
| scale model from GPT-4o-mini to GPT-4o | GPT-4o family · RecToM recommender belief | accuracy: 52.50 -> 74.74 | Improved |
| judgement task: binary yes/no strategy effectiveness | LLMs · RecToM judgement reasoning | accuracy: below 50.00 random guess | Hurt |
Closest papers by meaning
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.91)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.90)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.89)
- AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling (NeurIPS 2025 · similarity 0.89)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.89)
- Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
