Research map / Detection and segmentation / Open-set and few-shot detection
KaLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks
David Tschirschwitz, Volker Rodehorst
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Greedy correspondence solver instead of MGM or SHM (ablation) | KalOS · synthetic noise testbed (LVIS, TexBiG, VinDr-CXR) | filtered Rand Index / F1 | Improved |
| use Greedy solver instead of MGM and SHM for permutation stability (ablation) | KalOS · NuCLS | mean ARI: 0.9327 (SHM) / 0.9606 (MGM) -> 0.99998 | Improved |
| use Agglomerative Hierarchical Clustering as solver (ablation) | KalOS · synthetic noise testbed | filtered Rand Index | Hurt |
| use globally optimal MGM solver instead of Greedy (ablation) | KalOS · synthetic noise testbed | recall / F1 | Hurt |
| use class-aware cost psi_soft instead of localization-only cost psi_neg (ablation) | KalOS · synthetic noise testbed | filtered Rand Index / F1 | Improved |
| increase number of raters from 2 to 18 (ablation) | KalOS score · synthetic noise at fixed magnitude | mean KalOS score: diminishing returns beyond roughly 6-8 raters | Improved |
| split raters into two opposing collaboration clusters (e.g. 3-3 instead of 5-1) (ablation) | KalOS score · synthetic LVIS raters | KalOS score | Hurt |
| raise synthetic noise magnitude lambda from 1.0 to 5.0 (ablation) | noise generator · LVIS | signal-loss ratio: 4.2% -> 28.3% | Hurt |
| use three raters instead of one synthetic rater for solver recovery (ablation) | correspondence solvers · synthetic noise testbed | filtered Rand Index | Hurt |
Closest papers by meaning
- Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.89)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.86)
- COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts (CVPR 2025 · similarity 0.86)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.86)
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
