Research map / Label-efficient and robust learning / OOD and anomaly detection
Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ImageNet-ES, a 202k-image dataset captured with a real camera in a controllable light-and-sensor testbed (ES-Studio), and uses it to show that OOD detectors, augmentations, and sensor tuning behave differently under real covariate shifts than under digital perturbations.
Problem
Robustness and OOD benchmarks mostly rely on digital perturbations of already-acquired images, which do not capture distribution shifts arising in the physical image acquisition process (light and camera sensor settings).
Approach
Directly capture images of displayed Tiny-ImageNet samples with a physical camera while systematically varying ambient light and ISO, shutter speed, and aperture in a controlled dark studio, creating real covariate shifts rather than simulated ones; then use this to re-examine OOD definitions, detectors, domain generalization, and camera sensor control.
Key results
- Best-case sensor parameter tuning improves ResNet-50 on ImageNet-ES by 9.9 to 47.9 points over auto exposure, and EfficientNet-B0 with best settings outperforms OpenCLIP-h under auto exposure and all-params settings.
- Existing OOD detectors (MSP, ODIN, ReAct, ASH, ViM) show widely spread OOD scores on ImageNet-ES; ViM, the SOTA for semantic shift, performs worst on C-OOD F1 while best on S-OOD benchmarks.
- Adding environment/sensor augmentations to digital augmentations improves ResNet-50 accuracy on ImageNet-ES (e.g., 49.1 to 54.5 or 55.8) and on ImageNet-C (up to 57.9 mCE-style accuracy) in the reported table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add basic digital augmentations (ablation) | ResNet-50 · ImageNet-C | accuracy: 51.0 -> 51.7 | Helped |
| add basic digital augmentations (ablation) | ResNet-50 · ImageNet-ES | accuracy: 49.6 -> 50.4 | Helped |
| add advanced digital augmentations (ablation) | ResNet-50 · ImageNet-C | accuracy: 51.0 -> 57.4 | Helped |
| add advanced digital augmentations (ablation) | ResNet-50 · ImageNet-ES | accuracy: 49.6 -> 49.1 | Hurt |
| add ImageNet-ES environment and sensor augmentations (ablation) | ResNet-50 · ImageNet-ES | accuracy: 49.6 -> 55.8 | Helped |
| add ImageNet-ES environment and sensor augmentations (ablation) | ResNet-50 · ImageNet-C | accuracy: 51.0 -> 51.8 | Helped |
| use manual sensor parameters selected for highest accuracy instead of auto exposure | EfficientNet-B0 · ImageNet-ES | accuracy: 51.4 -> 83.8 | Helped |
| use manual sensor parameters selected for highest accuracy instead of auto exposure | ResNet-50 · ImageNet-ES | accuracy: 32.2 -> 80.1 | Helped |
| train with environmental and sensor perturbations in addition to digital augmentations (ablation) | ResNet-50 · ImageNet-C | accuracy: 51.0 -> 57.9 | Helped |
Surprising findings
- Aggressive digital augmentations (AugMix, DeepAugment) improve ImageNet-C robustness but degrade accuracy on real ImageNet-ES captures, contradicting prior work.
- Auto exposure degrades accuracy relative to original images for all models, and images preferred by humans do not necessarily yield the best model predictions.
Limitations
- Captures are of photos displayed on a screen rather than real 3D objects or printed photos, missing real object-light interactions.
- Sensor control handles only physical light and needs to be combined with digital post-processing.
- Training a neural network for sensor control would require gradients without extra photos.
Experimental setup
Datasets: Tiny-ImageNet ImageNet-ES ImageNet ImageNet-C Texture iNaturalist NINCO SSB-hard OpenImage-O
Benchmarks: ImageNet-ES ImageNet-C ImageNet Texture iNaturalist NINCO SSB-hard OpenImage-O
Metrics: classification accuracy F1 score FPR@95
Baselines: MSP ODIN ReAct ASH ViM DeepAugment AugMix
Builds on: MS-OOD DeepAugment AugMix ImageNet-C
Closest papers by meaning
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.88)
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox (NeurIPS 2024 · similarity 0.88)
- LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models (ICML 2025 · similarity 0.88)
- Image Background Serves as Good Proxy for Out-of-distribution Data (ICLR 2024 · similarity 0.87)
- Improving Out-of-Distribution Detection via Dynamic Covariance Calibration (ICML 2025 · similarity 0.86)
- Leveraging Perturbation Robustness to Enhance Out-of-Distribution Detection (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.