Research map / Detection and segmentation / Open-set and few-shot detection
Lean Crowdsourcing: Combining Humans and Machines in an Online System
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a worker-skill model (ablation) | Binary annotation crowdsourcing · Scorpions | error: reduced converged error by about 33% | Helped |
| Use online stopping to collect a variable number of annotations (ablation) | Binary annotation crowdsourcing · Scorpions | human annotations per image: roughly halved the number of annotations | Helped |
| Add calibrated computer-vision predictions to online crowdsourcing (ablation) | Binary annotation crowdsourcing · Scorpions | human annotations per image: reduced the number of annotations per image by an additional factor of 2.4 | Helped |
| Use calibrated computer-vision confidence estimates instead of treating vision as a worker (ablation) | Binary annotation crowdsourcing · Scorpions | human annotations per image: reduced annotations by a factor of 1.8 | Helped |
| Tighten the risk threshold from τǫ=.02 to τǫ=.005 (ablation) | Lean Crowdsourcing · Scorpions | error: 0.051 -> 0.041 | Helped |
| Tighten the risk threshold from τǫ=.02 to τǫ=.005 (ablation) | Lean Crowdsourcing · Scorpions | average human workers per image: 1.37 -> 1.93 | Hurt |
| Use a worker-skill model (ablation) | Bounding-box crowdsourcing · Pedestrian bounding boxes | error: reduced the error by a factor of 2 | Helped |
| Use online stopping to collect a variable number of annotations (ablation) | Bounding-box crowdsourcing · Pedestrian bounding boxes | human workers per image: by a factor of 1.7 without increasing error | Helped |
| Add computer vision to online crowdsourcing (ablation) | Bounding-box crowdsourcing · Pedestrian bounding boxes | human annotations: reduced annotation by an additional 29% | Helped |
| Use a worker-skill model (ablation) | Part-keypoint crowdsourcing · NABirds parts | error: 10% reduction in error | Helped |
| Use online stopping to collect a variable number of annotations (ablation) | Part-keypoint crowdsourcing · NABirds parts | annotation time: cut annotation time roughly in half | Helped |
| Use computer vision in the annotation loop (ablation) | Lean Crowdsourcing · Part-keypoint annotation | human annotations per part | No clear effect |
Closest papers by meaning
- Lean Multiclass Crowdsourcing (CVPR 2018 · similarity 0.93)
- Learning From Noisy Singly-labeled Data (ICLR 2018 · similarity 0.89)
- Towards Professional Level Crowd Annotation of Expert Domain Data (CVPR 2023 · similarity 0.87)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.86)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.85)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.