Research map / Label-efficient and robust learning / Recognition and long-tail
MetricOpt: Learning To Optimize Black-Box Evaluation Metrics
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MetricOpt with SGD to cross-entropy finetuning | CIFAR-10 classification · CIFAR-10 | MCR (%): 7.19±0.16 -> 6.63±0.05 | Helped |
| Use a jointly learned optimizer instead of SGD with MetricOpt (ablation) | MetricOpt + cross-entropy on CIFAR-10 · CIFAR-10 | MCR (%): 6.63±0.05 -> 6.58±0.06 | Helped |
| Add MetricOpt with SGD to cross-entropy finetuning | CIFAR-10 classification · CIFAR-10 | AUCPR (%): 90.8±0.18 -> 95.7±0.12 | Helped |
| Add MetricOpt with SGD to AUCPR loss | CIFAR-10 classification · CIFAR-10 | AUCPR (%): 94.2 -> 96.6±0.10 | Helped |
| Use a jointly learned optimizer instead of SGD with MetricOpt (ablation) | MetricOpt + cross-entropy on CIFAR-10 · CIFAR-10 | AUCPR (%): 95.7±0.12 -> 96.2±0.09 | Helped |
| Add MetricOpt to surrogate losses | A9A classification · A9A | MCR: 0.1502 -> 0.1369 | Helped |
| Add MetricOpt to surrogate losses | CoverType classification · CoverType | JAC: 0.7808 -> 0.7758 | Hurt |
| Transfer MetricOpt from CIFAR-10 to ImageNet | NASNet-A classification · ImageNet | Top-1 accuracy (%): 73.5 -> 74.4 | Helped |
| Add MetricOpt to Recall loss | Image retrieval · Stanford Online Products | Recall@1 (%): 78.6 -> 80.4 | Helped |
| Add MetricOpt to AP loss | Faster R-CNN · Pascal VOC 07 test | AP50: 75.7 -> 78.3 | Helped |
| Use ordinal embedding loss when training the value function (ablation) | MetricOpt for CIFAR-10 MCR · CIFAR-10 | value-function prediction error: 0.052 vs. 0.023 | Helped |
| Collect fewer metric observations, using K=2.5%T instead of the default K=5%T (ablation) | MetricOpt value-function training · CIFAR-10 | final optimized metric | Hurt |
Closest papers by meaning
- Optimizing Rank-Based Metrics With Blackbox Differentiation (CVPR 2020 · similarity 0.89)
- Optimizing Black-box Metrics with Iterative Example Weighting (ICML 2021 · similarity 0.87)
- Learning Low-Dimensional Metrics (NeurIPS 2017 · similarity 0.87)
- Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering (AAAI 2019 · similarity 0.86)
- A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction (AAAI 2026 · similarity 0.86)
- A General Online Algorithm for Optimizing Complex Performance Metrics (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.