Research map / Large language models / Language, safety and interpretability
ARTICLE: Annotator Reliability Through In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARTICLE, an in-context learning framework that flags inconsistent annotators by how well an LLM can predict each annotator's labels from their other annotations, then models group-level offense perception using only consistent annotators.
Problem
Estimating annotator quality on subjective tasks such as offensive speech detection, where outlier-based methods can mistake sincere minority perspectives for poor work and require multiple annotators per item.
Approach
Use self-consistency rather than divergence from majority: each annotator is modeled by an LLM with 10 of their own labeled examples as in-context demonstrations, and the F1 on their remaining labels serves as a consistency score; annotators below a threshold k are removed.
Key results
- On D_TR at k>=0.45, ARTICLE group-level F1 was 0.696 (Democrat) vs 0.669 for CrowdTruth at WQS>=0.6, with similar annotator retention (~70%).
- On VOICED, ARTICLE (k>=0.45) reached F1 0.616 (Democrat), 0.605 (Republican), 0.557 (Independent) vs 0.449, 0.435, 0.453 for CrowdTruth at WQS>=0.7.
- Jaccard similarity between inconsistent-annotator sets from different LLMs was 0.60-0.68, while LLM-vs-CrowdTruth similarity was 0.16-0.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ARTICLE rather than CT (WQS ≥ 0.6) | Group-level offense modeling · Toxicity Ratings (TR), Democrat | F1-score: 0.669 ± 0.016 -> 0.696 ± 0.015 | Helped |
| Use ARTICLE rather than CT (WQS ≥ 0.6) | Group-level offense modeling · Toxicity Ratings (TR), Republican | F1-score: 0.642 ± 0.018 -> 0.671 ± 0.017 | Helped |
| Use ARTICLE rather than CT (WQS ≥ 0.6) | Group-level offense modeling · Toxicity Ratings (TR), Independent | F1-score: 0.665 ± 0.018 -> 0.696 ± 0.017 | Helped |
| Use ARTICLE rather than CT (WQS ≥ 0.7) | Group-level offense modeling · VOICED, Democrat | F1-score: 0.449 ± 0.036 -> 0.616 ± 0.041 | Helped |
| Use ARTICLE rather than CT (WQS ≥ 0.7) | Group-level offense modeling · VOICED, Republican | F1-score: 0.435 ± 0.032 -> 0.605 ± 0.042 | Helped |
| Use ARTICLE rather than CT (WQS ≥ 0.7) | Group-level offense modeling · VOICED, Independent | F1-score: 0.453 ± 0.036 -> 0.557 ± 0.042 | Helped |
| Increase the annotator-consistency threshold k from 0 to 0.45 (ablation) | ARTICLE · Toxicity Ratings (TR) and VOICED, political-group offense modeling | F1-score | Helped |
| Increase k beyond 0.5 (ablation) | ARTICLE · Toxicity Ratings (TR) and VOICED, political-group offense modeling | F1-score | Mixed |
| Use different LLMs to identify inconsistent annotators | ARTICLE · Toxicity Ratings (TR) and VOICED | Jaccard similarity: ≥0.60 | No clear effect |
Surprising findings
- ARTICLE and CrowdTruth flag largely different annotators (Jaccard 0.16-0.20 on VOICED), suggesting they measure different aspects of quality.
- Performance plateaus or slightly declines for k above 0.5 rather than continuing to improve.
Limitations
- Relies on LLM self-consistency, which may carry model biases, especially for cultural or linguistic nuance.
- Cannot distinguish justified disagreement from genuine inconsistency.
- Effectiveness depends on the number of annotations available per annotator; only tested on offensive speech datasets.
Experimental setup
Datasets: Toxicity Ratings (D_TR) VOICED
Benchmarks: Toxicity Ratings VOICED
Metrics: F1-score Jaccard similarity percentage of annotators retained percentage of comments retained
Baselines: CrowdTruth (CT) GPT-3.5-turbo (as ARTICLE backbone) k=0 no-filtering baseline
Builds on: in-context learning CrowdTruth
Closest papers by meaning
- ARTICLE: Annotator Reliability Through In-Context Learning (Student Abstract) (AAAI 2025 · similarity 0.99)
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.85)
- Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models (ICLR 2024 · similarity 0.84)
- Collaborative Refining for Learning from Inaccurate Labels (NeurIPS 2024 · similarity 0.84)
- Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator (NeurIPS 2025 · similarity 0.84)
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.