Research map / Vision-language and multimodal / Human and embodied modeling
Consensus and Subjectivity of Skin Tone Annotation for ML Fairness
Candice Schumann, Gbolahan Oluwafemi Olanubi, Auriel Wright, Ellis Monk, Courtney Heldreth, Susanna Ricco
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use trained crowdsourced annotators rather than requiring subject-matter experts | MST skin-tone annotation · MST-E | inter-rater reliability (ICC) | Improved |
| Combine annotations from a geographically diverse pool | Crowdsourced MST annotation · MST-E subjects | within 1-point of oracle annotations (%): 94.7% | Improved |
| Annotate challenging images under poor lighting | Trained crowdsourced annotators · MST-E Challenging Images | inter-rater reliability (ICC): Golden Images ICC ranged from 0.9 to 0.96; Challenging Images ICC ranged from 0.82 to 0.95 | Hurt |
| Collect annotations in different geographic regions | MST skin-tone annotation · MST-E | regional differences in annotation distributions: Pillai's trace = 1.97, p<0.05 | Mixed |
| Annotate in-the-wild images rather than controlled images | Crowdsourced MST annotation · Open Images subset versus MST-E | inter-rater reliability (ICC): chi^2=0.26, p=0.99>0.05 | No clear effect |
| Use one annotator per region for high-quality images or consensus across multiple images of one person | MST skin-tone annotation · MST-E | inter-rater agreement | Improved |
| Use two annotators per region for challenging images | MST skin-tone annotation · MST-E Challenging Images | inter-rater agreement: 10 annotators | Improved |
Closest papers by meaning
- FACET: Fairness in Computer Vision Evaluation Benchmark (ICCV 2023 · similarity 0.89)
- Beyond Skin Tone: A Multidimensional Measure of Apparent Skin Color (ICCV 2023 · similarity 0.86)
- Mitigating Demographic Bias in Facial Datasets with Style-Based Multi-attribute Transfer (IJCV 2021 · similarity 0.86)
- AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark (CVPR 2025 · similarity 0.84)
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition (NeurIPS 2025 · similarity 0.83)
- FaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
