Research map / Label-efficient and robust learning / OOD and anomaly detection

LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models

ICML 2025 · Label-efficient and robust learning › OOD and anomaly detection · approach: Provable and feature-based OOD detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LAION-C, a six-distortion, 16-superclass OOD benchmark designed to be out-of-distribution even for web-scale training data, evaluated on 58 vision models, GPT-4o, Gemini 1.5 Pro and 19 human participants.

Problem

ImageNet-C corruptions (blur, noise, JPEG artifacts) are no longer out-of-distribution for models trained on web-scale datasets like LAION, and saturating scores on ImageNet-era OOD benchmarks make it unclear whether OOD generalization has truly improved.

Approach

Construct six synthetic, deliberately artificial distortions (Mosaic, Glitched, Vertical Lines, Geometric Shapes, Stickers, Luminance Checkerboard) at five intensity levels, calibrated so humans or ViT-B reach chance at the hardest level, and pair them with a 16-superclass human psychophysics baseline.

Key results

• The best model EVA-G-P14-560-M30M-IN22K reaches 67.5% average LAION-C accuracy versus 55.2% for the best human observer.
• Average model accuracy on ImageNet-C stays above or near 60%, while LAION-C shows wider inter-model variance (std about 27%).
• Fine-tuning ViT-H on LAION-C distortions raises accuracy substantially (e.g., Geometric Shapes from 64.4% to 89.8%), suggesting the task remains solvable.

Measured effects reported in the paper

Surprising findings

• Best models match or outperform the best human observers on LAION-C, a reversal from earlier human-superior OOD findings.
• Error consistency between models and humans ranges only about 0 to 0.4, indicating models use different strategies than humans despite high accuracy.

Limitations

• Does not explain why specific models underperform on specific distortions.
• Distortions are highly artificial and may not reflect natural real-world shifts.
• Human study uses a small pool of 19 university students and a 16-class simplification.

Experimental setup

Datasets: ImageNet validation set ImageNet-1K training set LAION-400M

Benchmarks: LAION-C ImageNet-C (16-class) ImageNet validation

Metrics: top-1 accuracy error consistency (Cohen's kappa) FID Kendall's tau

Baselines: ImageNet-C ResNet MobileNet ViT ConvNeXt EVA GPT-4o Gemini 1.5 Pro

Builds on: ImageNet-C error consistency Psychophysics Toolbox

Closest papers by meaning

• In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.89)
• LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.88)
• Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains (CVPR 2024 · similarity 0.88)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.87)
• ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object (CVPR 2024 · similarity 0.87)
• A Sober Look at the Robustness of CLIPs to Spurious Features (NeurIPS 2024 · similarity 0.86)