Research map / Label-efficient and robust learning / OOD and anomaly detection
What Variables Affect Out-of-Distribution Generalization in Pretrained Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits the tunnel effect hypothesis for pretrained DNN embeddings, finding it is not universal: higher resolution, more classes, augmentation and spatial reduction weaken it, and small-resolution toy datasets exacerbate it.
Problem
Whether the tunnel effect (intermediate neural collapse causing late layers to hurt OOD transfer) holds universally, and which variables (architecture, data, resolution, augmentation) control its strength, given prior work only used low-resolution toy datasets.
Approach
Defines three OOD-grounded metrics (% OOD performance retained, ID-OOD Pearson correlation, ID/OOD alignment) to quantify tunnel strength, and uses paired controlled experiments plus SHAP slope analysis over 64 backbones and thousands of linear probes to rank variable effects.
Key results
- Tunnel effect is not universal: of 64 DNNs, 4 had negligible, 8 weak, 13 medium and 39 strong tunnels by % OOD performance retained.
- Augmentation raised % OOD performance retained from 64.26% to 78.41% (p<0.001, medium effect); training on 32x32 vs 224x224 images with ID/OOD alignment showed large effect (|delta|=0.625).
- VGGm without early max-pool (spatial reduction 1.0) retained 84.40% OOD performance vs 64.85% for VGGm (p<0.001, large effect).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add random resized crop and random horizontal flip | ImageNet-100-trained models · OOD datasets (8-dataset average) | % OOD performance retained: 64.26% -> 78.41% | Helped |
| Increase image resolution from 32×32 to 64×64 (ablation) | ImageNet-100-trained models · ImageNet-100; OOD datasets | % OOD performance retained: negli.~(0.002); 0.315 | No clear effect |
| Increase image resolution from 32×32 to 128×128 or 224×224 (ablation) | ImageNet-100-trained models · ImageNet-100; OOD datasets | % OOD performance retained: small~(0.171); small~(0.198) | Helped |
| Omit max-pool in the first 2 stages (φ=0.5 to φ=1.0) (ablation) | VGGm · OOD datasets (8-dataset average) | % OOD performance retained: 64.85% -> 84.40% | Helped |
| Increase stem size from 3×3 to 7×7 (ablation) | VGGm-17 / ResNet-18 comparison · OOD datasets (8-dataset average) | % OOD performance retained: 76.74% -> 66.66% | Hurt |
| Increase stem size from 3×3 to 7×7 (ablation) | VGGm-17 / ResNet-18 comparison · OOD datasets (8-dataset average) | Pearson correlation | No clear effect |
| Increase depth from 11 to 17 layers (ablation) | VGGm · OOD datasets (8-dataset average) | % OOD performance retained: 89.19% -> 69.41% | Hurt |
| Increase over-parameterization level from γ=74.7 to γ=168.4 (ablation) | VGGm-11 / ResNet-34 comparison · OOD datasets (8-dataset average) | % OOD performance retained: 87.22% -> 62.78% | Hurt |
| Increase the number of classes while keeping training samples fixed at 10,000 (ablation) | VGGm-11 trained on ImageNet-100 · ImageNet-100; 10,000 training samples | Tunnel effect | Helped |
| Increase samples per class while holding the class count at 100 (ablation) | VGGm-11 trained on ImageNet-100 · ImageNet-100; 100 classes | Tunnel effect | No clear effect |
| Swap in task-2 tunnel T₂ for task-1 tunnel T₁ (ablation) | ResNet-18 continual learning · ImageNet-100 continual learning, 32×32 | 1st-task accuracy: 64.04 -> 37.64 | Hurt |
| Fine-tune E₂ together with T₁ instead of fine-tuning E₂ alone (ablation) | ResNet-18 continual learning · ImageNet-100 continual learning, 32×32 | 1st-task accuracy after fine-tuning: 45.60 -> 51.40 | Helped |
Surprising findings
- Widely used ImageNet-1K pretrained CNN and ViT backbones largely show no tunnel, except ResNet-50.
- Over-parameterization level plays a minor role compared with class count, resolution, and augmentation, contrary to the original tunnel effect work.
Limitations
- Only vision datasets and mainly supervised learning studied; SSL analysis limited to pretrained backbones without SHAP.
- Augmentation policies limited to random resized crop and horizontal flip in controlled experiments.
- No theoretical framework explaining the tunnel effect is provided.
Experimental setup
Datasets: ImageNet-100 CIFAR-10 CIFAR-100 ImageNet-1K NINCO ImageNet-R Oxford 102 Flowers CUB-200 Aircrafts Oxford-IIIT Pets STL-10
Benchmarks: ImageNet-100 CIFAR-10 CIFAR-100 ImageNet-1K NINCO ImageNet-R Oxford 102 Flowers CUB-200 Aircrafts Oxford-IIIT Pets STL-10
Metrics: % OOD performance retained Pearson correlation ID/OOD alignment linear probe accuracy Cliff's delta SHAP slope
Baselines: tunnel effect hypothesis (NeurIPS 2023) setup
Builds on: tunnel effect hypothesis (NeurIPS 2023) intermediate neural collapse SHAP
Closest papers by meaning
- Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning (ICML 2025 · similarity 0.90)
- Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize (ICML 2024 · similarity 0.87)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.87)
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. (NeurIPS 2024 · similarity 0.86)
- Interpretability Illusions in the Generalization of Simplified Models (ICML 2024 · similarity 0.86)
- Data Distributional Properties As Inductive Bias for Systematic Generalization (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.