Research map / Large language models / Language, safety and interpretability
ARTICLE: Annotator Reliability Through In-Context Learning (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARTICLE, an in-context learning framework that flags inconsistent annotators via per-annotator self-consistency (Mistral-7B ICL F1) and then models group-level offense perception after removing them.
Problem
Distinguishing poor-quality annotators from sincere annotators with differing opinions on subjective tasks such as offensive speech detection, where outlier-based quality assessment can erase minority perspectives.
Approach
Estimate annotator quality via self-consistency: model each annotator individually with ICL on a small training set and treat low predictive F1 on their remaining labels as a sign of inconsistency, removing annotators below threshold k before aggregating group labels.
Key results
- On both datasets, the F1 score with threshold k=0.45 surpassed the no-filtering baseline (k=0) for almost all political groups (Democrat, Republican, Independent).
- F1 generally rises as k increases up to about 0.5, after which performance plateaus or slightly declines.
- Qualitative examples show ARTICLE flags annotators whose labels contradict on similar content, such as labeling a racial slur as non-offensive.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Set the inconsistency threshold to k = 0.45 and remove annotators whose individual F1-score is below it | ARTICLE with Mistral-7B · DTR and DVOICED, Democrat, Republican, and Independent groups | F1 score | Helped |
| Increase the inconsistency threshold above k = 0.5 | ARTICLE with Mistral-7B · DTR and DVOICED, political groups | F1 score | Mixed |
Surprising findings
- Performance gains plateau or decline for k above 0.5, so more aggressive filtering is not always beneficial.
Limitations
- Only one LLM (Mistral-7B) is used, and the abstract mentions multiple LLMs without detail in the body provided.
- Evaluation is limited to two English offensive speech datasets and three political groups.
- Each annotator is modeled with only 10 training examples, which may make the consistency estimate noisy.
Experimental setup
Datasets: Toxicity Ratings (DTR) VOICED (DVOICED)
Benchmarks: Toxicity Ratings VOICED
Metrics: F1 score
Baselines: k=0 no-filtering baseline
Builds on: self-consistency Dawid-Skene annotator model
Closest papers by meaning
- ARTICLE: Annotator Reliability Through In-Context Learning (AAAI 2025 · similarity 0.99)
- Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models (ICLR 2024 · similarity 0.84)
- GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition (AAAI 2025 · similarity 0.84)
- Collaborative Refining for Learning from Inaccurate Labels (NeurIPS 2024 · similarity 0.84)
- Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator (NeurIPS 2025 · similarity 0.84)
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.