Research map / Label-efficient and robust learning / OOD and anomaly detection
Agreement-on-the-line: Predicting the Performance of Neural Networks under Distribution Shift
Christina Baek, Yiding Jiang, Aditi Raghunathan, J Zico Kolter
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate OOD accuracy by solving for model accuracies using OOD pairwise agreement (ALine-D) | Neural-network classifiers · CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C-Fog, CIFAR-10C-Snow, CIFAR-10C-Saturate, fMoW-wilds, RxRx1-wilds | MAE (%): 1.11; 3.93; 2.06; 1.45; 1.32; 0.41; 1.30; 0.27 | Improved |
| Estimate OOD accuracy by solving for model accuracies using pairwise agreement (ALine-D) | Neural-network classifiers · Camelyon17-wilds and iWildCam-wilds | MAE (%): 5.47; 4.95 | Mixed |
| Use agreement-line slope and bias to transform ID accuracy without directly using the target model's OOD agreement (ALine-S) | Neural-network classifiers · CIFAR-10.1, CIFAR-10.2, ImageNetV2, CIFAR-10C-Fog, CIFAR-10C-Snow, CIFAR-10C-Saturate, fMoW-wilds, RxRx1-wilds | MAE (%): 1.17; 3.93; 2.08; 1.75; 1.97; 0.77; 1.44; 0.52 | Mixed |
| Use ALine-D rather than ALine-S | OOD accuracy prediction · CIFAR-10.1, ImageNetV2, CIFAR-10C-Fog, CIFAR-10C-Snow, CIFAR-10C-Saturate, fMoW-wilds, RxRx1-wilds | MAE (%): 1.17 -> 1.11; 2.08 -> 2.06; 1.75 -> 1.45; 1.97 -> 1.32; 0.77 -> 0.41; 1.44 -> 1.30; 0.52 -> 0.27 | Improved |
| Use ALine-D rather than ATC | OOD accuracy prediction · ImageNetV2 and CIFAR-10C-Snow | MAE (%): 2.06 vs 1.14; 1.32 vs 1.31 | Hurt |
| Use ALine-D rather than ProjNorm | ResNet18 OOD accuracy estimates · CIFAR-10C | R²: 0.973 -> 0.974 | Improved |
| Use agreement-line estimation on a single training trajectory rather than independently trained models | ResNet18 on CIFAR-10 and ResNet50 on ImageNet · CIFAR-10.1 and ImageNetV2 | MAE (%): 2.19%; 3.61% | Mixed |
| Use more models in the ALine-D model set (ablation) | ALine-D · CIFAR-10.1 | MAE (%) | Improved |
| Use more models in the ALine-D model set (ablation) | ALine-D · ImageNetV2 | MAE (%): 2.5% | Improved |
| Use more models in the ALine-D model set (ablation) | ALine-D · CIFAR-10C Fog | MAE (%): below 1.8% | No clear effect |
| Use a diverse-architecture model set instead of models with one architecture (ablation) | ALine-D · CIFAR-10.1 and CIFAR-10C Fog | MAE (%) | Improved |
| Use a diverse-architecture model set instead of models with one architecture (ablation) | ALine-D · fMoW-wilds | MAE (%) | Hurt |
Closest papers by meaning
- Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.87)
- Predicting the Performance of Foundation Models via Agreement-on-the-Line (NeurIPS 2024 · similarity 0.87)
- Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line (NeurIPS 2024 · similarity 0.87)
- Position: Supervised Classifiers Answer the Wrong Questions for OOD Detection (ICML 2025 · similarity 0.85)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.84)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
