Research map / Detection and segmentation / Open-set and few-shot detection
How Trustworthy are Performance Evaluations for Basic Vision Tasks?
Tran Thien Dat Nguyen, Hamid Rezatofighi, Ba‐Ngu Vo, Ba-Tuong Vo, Silvio Savarese, Ian D. Reid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average F1 over the whole IoU threshold range (M-full) instead of IoU 0.5 (ablation) | F1 · Single-class detection sanity test | ranking error: 10.0 -> 2.15 | Improved |
| average mAP over IoU 0.5-0.95 (M-partial) instead of IoU 0.5 (ablation) | mAP · Multi-class detection sanity test | ranking error: 10.0 -> 7.52 | Improved |
| use optimal IoU threshold instead of IoU 0.5 (ablation) | mAP · Multi-class detection sanity test | ranking error: 10.0 -> 7.08 | Improved |
| average mAP over the whole IoU range (M-full) instead of IoU 0.5 (ablation) | mAP · Multi-class detection sanity test | ranking error: 10.0 -> 3.62 | Improved |
| replace F1 at IoU 0.5 with OSPA metric | F1 · Single-class detection sanity test | ranking error: 10.0 -> 1.97 | Improved |
| replace mAP at IoU 0.5 with OSPA metric | mAP · Multi-class detection sanity test | ranking error: 10.0 -> 1.86 | Improved |
| replace MOTA at IoU 0.5 with OSPA metric | MOTA · Multi-object tracking sanity test | ranking error: 5.18 -> 0.518 | Improved |
| use Hausdorff metric instead of OSPA | OSPA · Single-class detection sanity test | ranking error: 1.97 -> 17.8 | Hurt |
| use EMD metric instead of OSPA | OSPA · Multi-class detection sanity test | ranking error: 1.86 -> 2.80 | Hurt |
| use approximate truth reference instead of ground truth reference (ablation) | OSPA (IoU) · Tracking | ranking error: 0.536 -> 0.611 | No clear effect |
| use approximate truth reference instead of ground truth reference (ablation) | EMD (IoU) · Multi-class detection | ranking error: 4.16 -> 4.48 | No clear effect |
| use un-normalized OSPA (remove division by object count) (ablation) | OSPA · scenarios 1-10 dislocation test | prediction error | Hurt |
Closest papers by meaning
- Learning to Evaluate Perception Models Using Planner-Centric Metrics (CVPR 2020 · similarity 0.88)
- Playing for Benchmarks (ICCV 2017 · similarity 0.85)
- Object-Proposal Evaluation Protocol is 'Gameable' (CVPR 2016 · similarity 0.85)
- Evaluating Multiview Object Consistency in Humans and Image Models (NeurIPS 2024 · similarity 0.84)
- How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection (AAAI 2024 · similarity 0.84)
- Size-invariance Matters: Rethinking Metrics and Losses for Imbalanced Multi-object Salient Object Detection (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
