Research map / Learning theory and algorithms / Prediction and decision losses
Instance-Level Costs for Nuanced Classifier Evaluation
Kabir Kang, Stephen Mussmann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune RoBERTa end-to-end instead of frozen embeddings | RoBERTa on Jigsaw · Jigsaw | NEC (%): 3.29% -> 1.78% | Improved |
| fine-tune ResNet50 instead of frozen embeddings | ResNet50 on Turkey · Turkey | NEC (%): 4.16% -> 2.27% | Improved |
| add |Delta|-weighting to cross-entropy with frozen embeddings (ablation) | ResNet50-LP · iNaturalist | NEC (%): 10.39% -> 11.08% | Hurt |
| add |Delta|-weighting to cross-entropy with fine-tuning (ablation) | ResNet50-FT · iNaturalist | NEC (%): 9.34% -> 8.77% | Improved |
| add |Delta|-weighting to cross-entropy (ablation) | ResNet50-LP · Turkey | NEC (%): 4.16 -> 3.55 | Improved |
| add |Delta|-weighting to cross-entropy (ablation) | TF-IDF · Jigsaw | NEC (%): 1.76 -> 1.79 | No clear effect |
| add |Delta|-weighting on linearly predictable costs (ablation) | LogReg · Synthetic | NEC (%): 0.50 -> 0.40 | Improved |
| train with Delta-regression instead of classification (ablation) | LogReg · Synthetic | NEC (%): 0.50% -> 0.26% | Improved |
| use focal loss gamma=2 (ablation) | LogReg · Synthetic | NEC (%): 0.50% -> 17.99% | Hurt |
| use focal loss gamma=2 or 3 (ablation) | TF-IDF · Jigsaw | NEC (%): 1.76 -> 97.34 | Hurt |
| add Elkan Bayes-optimal thresholding (ablation) | Standard CE · Jigsaw | NEC (%): 1.76% -> 1.77% | No clear effect |
| add Elkan Bayes-optimal thresholding (ablation) | Standard CE · Turkey | NEC (%): 4.16% -> 3.67% | Improved |
Closest papers by meaning
- Learning Only When It Matters: Cost-Aware Long-Tailed Classification (AAAI 2024 · similarity 0.86)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.84)
- Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget (ICML 2024 · similarity 0.84)
- Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective (AAAI 2024 · similarity 0.84)
- The VOROS: Lifting ROC Curves to 3D to Summarize Unbalanced Classifier Performance (AAAI 2025 · similarity 0.83)
- Accuracy vs. Accuracy: Computational Tradeoffs Between Classification Rates and Utility (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
