Research map / Learning theory and algorithms / Prediction and decision losses
Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics
Matthias Kummerer, Thomas S. A. Wallis, Matthias Bethge
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the metric-specific optimal saliency map instead of one saliency map for all metrics | saliency models (AIM, BMS, eDN, OpenSALICON, SalGAN, DeepGaze II) · MIT-style fixation data, seven metrics | AUC, sAUC, NSS, IG, CC, KL-Div, SIM | Improved |
| evaluate with saliency maps derived from the model density rather than original maps | all six saliency models · seven saliency metrics | consistency of model rankings across metrics | Improved |
| use a map derived for another metric (instead of the matching one) (ablation) | saliency models · seven saliency metrics | metric performance | Hurt |
| evaluate log-density maps (prior unification) instead of metric-specific maps (ablation) | saliency models · NSS, CC, KL-Div, SIM | metric performance and ranking agreement | Hurt |
| use CC saliency map instead of NSS saliency map for AIM (ablation) | AIM · NSS metric | NSS | Mixed |
| use the SIM-specific map instead of the CC (blurred density) map (ablation) | saliency models · SIM metric | SIM: average difference 0.006 | No clear effect |
| add 8-bit binning to the raw density saliency map (ablation) | DeepGaze II · AUC | AUC: 0.88355 -> 0.87931 | Hurt |
| add 8-bit binning to the raw density saliency map (ablation) | AIM · AUC | AUC: 0.82883 -> 0.82855 | Hurt |
| equalize the histogram to be uniform before 8-bit binning | DeepGaze II · AUC | AUC under 8-bit binning: 0.87931 -> 0.88351 | Improved |
| equalize the histogram to be uniform before 8-bit binning | OpenSALICON · AUC | AUC under 8-bit binning: 0.86200 -> 0.86347 | Improved |
| use mean empirical saliency map instead of mean normalized empirical map for CC (ablation) | CC saliency map · simulated fixations | CC | No clear effect |
| compute SIM map with a different number of fixations per sample than the test data (ablation) | SIM saliency map · simulated fixations | SIM | Hurt |
Closest papers by meaning
- Saliency Prediction in the Deep Learning Era: Successes and Limitations (TPAMI 2019 · similarity 0.90)
- Modeling Saliency Dataset Bias (ICCV 2025 · similarity 0.90)
- Sanity Checks for Saliency Metrics (AAAI 2020 · similarity 0.89)
- Learning Saliency From Single Noisy Labelling: A Robust Model Fitting Perspective (TPAMI 2020 · similarity 0.88)
- What Do Different Evaluation Metrics Tell Us About Saliency Models? (TPAMI 2018 · similarity 0.88)
- Advancing Saliency Ranking with Human Fixations: Dataset Models and Benchmarks (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
