Research map / Image generation / Autoregressive and latent generation
Quality Assured: Rethinking Annotation Strategies in Imaging AI
Tim Rädsch, Annika Reinke, Vivienn Weru, Minu D. Tizabi, Nicholas Heller, Fabian Isensee, Annette Kopp-Schneider, Lena Maier-Hein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use annotation companies instead of MTurk crowdworkers | annotation pipeline · HeiCo instance segmentation | annotation cost relative to MTurk: 61.1% cheaper in the median; 24.5% to 81.5% of MTurk costs | Improved |
| use annotation companies instead of MTurk crowdworkers | annotation pipeline · HeiCo instance segmentation | spam annotations: no spam vs MTurk ~20% | Improved |
| use annotation companies instead of MTurk crowdworkers | annotation pipeline · HeiCo instance segmentation | DSC, NSD and severe errors | Improved |
| add internal QA by annotation company workers (ablation) | annotation companies, minimal text instructions · HeiCo instance segmentation | median DSC | No clear effect |
| add internal QA by annotation company workers (ablation) | annotation companies, extended text instructions · HeiCo instance segmentation | median DSC: two companies no change, two below 2% relative improvement | No clear effect |
| add internal QA by annotation company workers (ablation) | Company 2, minimal text instructions · HeiCo instance segmentation | fraction of images with severe errors: 0.22 -> 0.28 | Hurt |
| add internal QA by annotation company workers (ablation) | Company 3, extended text instructions · HeiCo instance segmentation | fraction of images with severe errors: 0.25 -> 0.17 | Improved |
| add internal QA by annotation company workers (ablation) | Company 4, all instruction types · HeiCo instance segmentation | fraction of images with severe errors: 0.09 -> 0.09 with extended text + pictures | No clear effect |
| improve labeling instructions (add extended text and pictures) instead of adding QA (ablation) | annotation companies · HeiCo instance segmentation | DSC and severe errors | Improved |
| apply QA to images with underexposure, intersecting or occluded objects (ablation) | internal QA · HeiCo instance segmentation, generalizable image characteristics | odds of improvement | Improved |
| apply QA to images with domain-specific characteristics (ablation) | internal QA · HeiCo instance segmentation, domain-specific image characteristics | odds of improvement | No clear effect |
| choose a more expensive or different-region annotation company | internal QA · HeiCo instance segmentation | QA quality improvement | No clear effect |
Closest papers by meaning
- AI-generated Image Quality Assessment in Visual Communication (AAAI 2025 · similarity 0.84)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.83)
- A-Bench: Are LMMs Masters at Evaluating AI-generated Images? (ICLR 2025 · similarity 0.82)
- ARTICLE: Annotator Reliability Through In-Context Learning (AAAI 2025 · similarity 0.82)
- FedA3I: Annotation Quality-Aware Aggregation for Federated Medical Image Segmentation against Heterogeneous Annotation Noise (AAAI 2024 · similarity 0.82)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
