Research map / Label-efficient and robust learning / Recognition and long-tail
Intelligent Calibration for Bias Reduction in Sentiment Corpora Annotation Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selecting a small, learned calibration set of reviews shown to annotators before the main sentiment labeling batch to reduce anchoring bias, validated with 1640 crowdworkers on Prolific and AMT.
Problem
Sequential sentiment annotation of reviews is subject to anchoring bias, where extreme early reviews distort subsequent scores and increase annotation error.
Approach
Use a small set of k=3 carefully chosen calibration reviews (selected via interpretable rules extracted from a logistic regression model over behavioral and ground-truth features) labeled before the main batch, instead of random or no calibration.
Key results
- Calibrated annotation achieved MAE 0.47 vs 0.58 non-calibrated and 0.54 random calibration on FOODA (19% and 13% reductions).
- On FOODB, calibration reduced MAE from 0.44 to 0.34 (22.7% reduction).
- On the hotel domain, calibration reduced MAE from 0.21 to 0.16 (24% improvement), transferring rules from food reviews.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3 calibration reviews selected with learned guidelines instead of no calibration | Sentiment annotation · FOODA | MAE: 0.58 -> 0.47 | Helped |
| Use 3 calibration reviews selected with learned guidelines instead of random calibration reviews | Sentiment annotation · FOODA | MAE: 0.54 -> 0.47 | Helped |
| Apply the selected 3-review calibration set instead of no calibration | Sentiment annotation · FOODB | MAE: 0.44 -> 0.34 | Helped |
| Transfer food-domain calibration guidelines and use 3 calibration reviews instead of no calibration | Sentiment annotation · HOTELS | MAE: 0.21 -> 0.16 | Helped |
| Present extreme positive or negative reviews first rather than a different initial review set | Sequential sentiment annotation · FOODA | MAE: null | Mixed |
Surprising findings
- Even random calibration sets can produce higher MAE than extreme-review sets, and MAE magnitude does not strictly align with average scores.
Limitations
- Calibration set size k=3 was set arbitrarily; trade-off with overhead not studied.
- Logistic regression classifier has modest performance (61% accuracy, ROC AUC 0.637).
- Requires an already labeled reliable review repository to select calibration reviews.
Experimental setup
Datasets: Amazon Fine Food Reviews (Kaggle) Hotel reviews (Kaggle, Datafiniti)
Benchmarks: FOODA15 food reviews FOODB15 food reviews HOTEL15 hotel reviews
Metrics: mean absolute error ROC AUC accuracy recall precision
Baselines: non-calibrated annotation random calibration set
Builds on: anchoring effect
Closest papers by meaning
- DanceHA: A Multi-Agent Framework for Document-Level Aspect-Based Sentiment Analysis (AAAI 2026 · similarity 0.84)
- ARTICLE: Annotator Reliability Through In-Context Learning (AAAI 2025 · similarity 0.84)
- Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations (AAAI 2024 · similarity 0.84)
- ARTICLE: Annotator Reliability Through In-Context Learning (Student Abstract) (AAAI 2025 · similarity 0.83)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.82)
- Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.