Research map / Large language models / Language, safety and interpretability
Understanding Annotator Perception: Modeling Psychological Inference from First- and Third-Person Annotations (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares third-person (crowd) annotations of psychological appraisals, emotions and intentions in consumer reviews against first-person writer self-reports, and fine-tunes BERT/RoBERTa and multi-task models to predict first-person labels from third-person ratings.
Problem
Third-person annotations of psychological states in text, often used as training labels for LLMs, may not match the writers' actual first-person psychological states.
Approach
Pairs first-person self-annotations of 1,400 purchase reviews with 365 crowd workers' third-person ratings, then measures the correspondence and trains BERT/RoBERTa classifiers and multi-task models to quantify how well third-person inferences predict first-person labels.
Key results
- Third-person BERT classifiers consistently underperform first-person BERT models; e.g., goal conduciveness and intent to promote show large F1 gaps (first-person 0.71 and 0.66 vs third-person best ~0.58-0.6 range).
- Pleasantness first-person F1 0.74 vs 0.56 (BERT) and 0.58 (RoBERTa) for third-person.
- Multi-task model using third-person appraisals and emotions predicts first-person emotion F1 of 0.64, with accuracy drops of 6-11% versus first-person framework.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train classifiers using third-person annotations rather than first-person annotations | BERT and RoBERTa classifiers · Pleasantness | F1: 0.74 compared to 0.56 and 0.58 | Hurt |
| Train classifiers using third-person annotations rather than first-person annotations | BERT and RoBERTa classifiers · Goal conduciveness | F1: 0.58; 70.5 / 0.71 | Hurt |
| Train classifiers using third-person annotations rather than first-person annotations | BERT and RoBERTa classifiers · Intent to promote | F1: 0.56; 72.4 / 0.66 | Hurt |
| Use appraisals as an intermediate task (ablation) | Multi-task model predicting intent to promote · Intent to promote | Accuracy: 63.3 | Helped |
| Use emotions as an intermediate task (ablation) | Multi-task model predicting intent to promote · Intent to promote | Accuracy: 62.6 | Helped |
| Use both appraisals and emotions as intermediate tasks (ablation) | Multi-task model predicting intent to promote · Intent to promote | Accuracy: 64.0 | Helped |
| Use appraisals as an intermediate task (ablation) | Multi-task model predicting intent to repurchase · Intent to repurchase | Accuracy: 61.9 | Helped |
| Use emotions as an intermediate task (ablation) | Multi-task model predicting intent to repurchase · Intent to repurchase | Accuracy: 62.6 | Helped |
| Use both appraisals and emotions as intermediate tasks (ablation) | Multi-task model predicting intent to repurchase · Intent to repurchase | Accuracy: 62.6 | Helped |
| Use third-person labels instead of first-person labels | Models predicting first-person variables · Intentions to promote and repurchase | Accuracy: 6-11% | Hurt |
Surprising findings
- Expectedness is better inferred from third-person than first-person annotations.
- Third-person and first-person ratings correspond at most moderately; behavioral intentions were inferred better than cognitive appraisals (r = .47).
Limitations
- Single domain (purchase reviews); future work proposes other social media domains.
- Third-person correspondence with first-person ratings is at most moderate.
- Some appraisals have high missing data rates, suggesting they are poorly conveyed in text.
Experimental setup
Datasets: PEACE-Reviews
Benchmarks: PEACE-Reviews
Metrics: accuracy F1 correlation
Baselines: BERT first-person models (Yeo and Jaidka 2023) RoBERTa first-person label framework (Yeo, Furniturewala, and Jaidka 2024)
Builds on: PEACE-Reviews multi-task framework of Yeo, Furniturewala, and Jaidka (2024)
Closest papers by meaning
- Uncovering Gaps in How Humans and LLMs Interpret Subjective Language (ICLR 2025 · similarity 0.87)
- The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans (NeurIPS 2026 · similarity 0.87)
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.85)
- OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling (NeurIPS 2026 · similarity 0.85)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.85)
- StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.