Research map / Probabilistic and scientific modeling / Molecules and proteins
Identifying and Benchmarking Natural Out-of-Context Prediction Problems
David Madras, Richard Zemel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use environment-level rather than label-level reweighting | ResNet-50 classifiers · NOOCh-CE, average of 12 tasks | AUC on hard examples: 0.870 -> 0.905 | Improved |
| Use environment-level rather than label-level reweighting | ResNet-50 classifiers · NOOCh-Gist, average of 12 tasks | AUC on hard examples: 0.830 -> 0.849 | Improved |
| Use GDRO instead of ERM | ResNet-50 classifiers · Waterbirds | AUC-Hard: 0.691 -> 0.929 | Improved |
| Evaluate on gist-defined rather than CE-defined challenge sets | robust learning methods · NOOCh average of 12 tasks | AUC on hard examples: 0.911 -> 0.849 | Hurt |
| Use environment-based information for training | robust learning methods · NOOCh-CE and NOOCh-Gist | AUC on hard examples: 0.865 -> 0.911; 0.826 -> 0.849 | Improved |
| Add data augmentation to ERM (ablation) | ERM · NOOCh | performance | No clear effect |
| Add an auxiliary prediction head for the highest-alpha context variable (ablation) | ERM · NOOCh | performance | No clear effect |
| Train ERM from scratch instead of using pretrained features (ablation) | ERM · NOOCh | performance | Hurt |
| Add adaptive affine transformations after each ResNet block (ablation) | ERM · NOOCh hard examples | performance | Improved |
| Use logistic regression on pretrained ImageNet features instead of finetuned ERM (ablation) | ResNet-50 pretrained features · NOOCh | performance | Hurt |
Closest papers by meaning
- OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization (CVPR 2022 · similarity 0.85)
- Deep Learning Through the Lens of Example Difficulty (NeurIPS 2021 · similarity 0.85)
- Contextual Reliability: When Different Features Matter in Different Contexts (ICML 2023 · similarity 0.84)
- Can In-context Learning Really Generalize to Out-of-distribution Tasks? (ICLR 2025 · similarity 0.84)
- Understanding the Role of Training Data in Test-Time Scaling (ICLR 2026 · similarity 0.83)
- Context is Environment (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
