Research map / Probabilistic and scientific modeling / Molecules and proteins
False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use knockoff filter via ISS for annotator selection | Knockoff ISS · Simulation, 150 annotators | actual FDR: 0.0959 to 0.1229 around q=10% | Helped |
| Use ISS instead of LASSO in the knockoff filter | Knockoff ISS · Simulation | actual FDR and true discoveries: 0.0959 vs 0.0711 | No clear effect |
| Raise good-annotator error p1 from 10% to 40% (ablation) | Knockoff ISS · Simulation, p2=40% | number of true discoveries: 49.95 -> 49.75 | Hurt |
| Raise good-annotator error p1 from 10% to 40% (ablation) | Knockoff LASSO · Simulation, p2=40% | number of true discoveries: 49.95 -> 49.85 | Hurt |
| Raise bias probability p2 from 40% to 50% (ablation) | Knockoff ISS · Simulation, p1=10% | number of true discoveries: 49.95 -> 50.00 | Helped |
| Apply correction step (re-estimate after removing detected annotators) | Knockoff · Human age dataset | agreement with groundtruth ranking | Helped |
| Use LASSO vs ISS in knockoff on real datasets | Knockoff · Human age dataset | abnormal annotators detected: 17 users | No clear effect |
| Use LASSO vs ISS in knockoff on real datasets | Knockoff · Reading level dataset | abnormal annotators detected: 6 annotators | No clear effect |
| Use LASSO vs ISS in knockoff on WorldCollege | Knockoff · WorldCollege | abnormal annotators detected: 36 annotators | No clear effect |
| Use knockoff detection instead of a simple left/right click ratio threshold | Knockoff ISS · Reading level dataset | detecting position bias: MR 3/6 vs 5/6 | Helped |
Closest papers by meaning
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.86)
- Label Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model (NeurIPS 2023 · similarity 0.86)
- Learning from Crowds by Modeling Common Confusions (AAAI 2021 · similarity 0.85)
- MA$^3$S: Model-Agnostic Active Annotation Strategy for Crowdsourcing (ICML 2026 · similarity 0.85)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.85)
- IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.