AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Consensus and Subjectivity of Skin Tone Annotation for ML Fairness

Candice Schumann, Gbolahan Oluwafemi Olanubi, Auriel Wright, Ellis Monk, Courtney Heldreth, Susanna Ricco

NeurIPS 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Multimodal and affective interaction modeling · 14 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use trained crowdsourced annotators rather than requiring subject-matter expertsMST skin-tone annotation · MST-Einter-rater reliability (ICC)Improved
Combine annotations from a geographically diverse poolCrowdsourced MST annotation · MST-E subjectswithin 1-point of oracle annotations (%): 94.7%Improved
Annotate challenging images under poor lightingTrained crowdsourced annotators · MST-E Challenging Imagesinter-rater reliability (ICC): Golden Images ICC ranged from 0.9 to 0.96; Challenging Images ICC ranged from 0.82 to 0.95Hurt
Collect annotations in different geographic regionsMST skin-tone annotation · MST-Eregional differences in annotation distributions: Pillai's trace = 1.97, p<0.05Mixed
Annotate in-the-wild images rather than controlled imagesCrowdsourced MST annotation · Open Images subset versus MST-Einter-rater reliability (ICC): chi^2=0.26, p=0.99>0.05No clear effect
Use one annotator per region for high-quality images or consensus across multiple images of one personMST skin-tone annotation · MST-Einter-rater agreementImproved
Use two annotators per region for challenging imagesMST skin-tone annotation · MST-E Challenging Imagesinter-rater agreement: 10 annotatorsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy