Research map / Label-efficient and robust learning / OOD and anomaly detection
Intermediate Layer Classifiers for OOD generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that frozen intermediate-layer representations, probed with linear Intermediate Layer Classifiers (ILCs), generalize better OOD than penultimate-layer last-layer retraining, in both few-shot and zero-shot settings.
Problem
Last-layer retraining on penultimate features is commonly used for OOD generalization under distribution shift, but it is unclear whether the penultimate layer holds the most useful features for OOD transfer.
Approach
Train linear probes (Intermediate Layer Classifiers) on frozen intermediate layers of pretrained networks instead of only the last layer, selecting the best layer on an OOD validation set; intermediate representations are found to be less sensitive to distribution shift than penultimate features, enabling zero-shot OOD gains using only ID training data.
Key results
- Few-shot OOD accuracy with ResNet improves over last-layer retraining by +16.8 pp on CMNIST, +6.3 on CIFAR-10C, +1.6 on CIFAR-100C, +6.3 on MultiCelebA.
- Zero-shot worst-group accuracy on Waterbirds rises from 79.4% (last layer, ID data) to 87.1% (best ILC), and on CelebA from 56.0% to 82.0%.
- On CIFAR-10C, all models gain roughly +2 to +5 pp mean accuracy using the best intermediate layer instead of the last layer, with ViT reaching about 71% average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the best intermediate-layer classifier instead of last-layer retraining | ResNet OOD classifiers · CMNIST, CIFAR-10, CIFAR-100C, MultiCelebA | accuracy: (+16.8, +6.3, +1.6, +6.3) percentage points | Helped |
| Use the best intermediate-layer classifier instead of last-layer retraining | ViT OOD classifiers · CMNIST, CIFAR-10C, CIFAR-100C | accuracy: (+1.1, +1.3,-0.2) percentage points | Mixed |
| Use ILCs rather than last-layer retraining with a small OOD probe set ($\pi \leq 0.03$) | OOD classifiers on subpopulation shifts · Waterbirds, CelebA, MultiCelebA | accuracy: (+5.7, +3.4, +27.0) percentage points | Helped |
| Use ILCs rather than last-layer retraining at $\pi=0.25$ | OOD classifiers on input shifts · CMNIST, CIFAR-10C, CIFAR-100C | accuracy: (+12, +3.6, +1.0) percentage points | Helped |
| Train an intermediate-layer classifier on ID data instead of retraining the last layer on ID data | Zero-shot OOD classifier · Waterbirds | WGA: 79.4%→87.1% | Helped |
| Train an intermediate-layer classifier on ID data instead of retraining the last layer on ID data | Zero-shot OOD classifier · CelebA | WGA: 56.0%→82.0% | Helped |
| Train an intermediate-layer classifier on ID data instead of retraining the last layer on ID data | Zero-shot OOD classifier · MultiCelebA | WGA: 20.8%→46.0% | Helped |
| Use ILCs instead of last-layer retraining | Zero-shot OOD classifier · CIFAR-10C | mean accuracy: +2\%p to +5\%p | Helped |
| Use ILCs instead of last-layer retraining | ViT zero-shot classifier · CIFAR-10C | average accuracy: 71% | Helped |
| Use ILCs instead of last-layer retraining | OOD classifiers · CIFAR-100C | accuracy: +3%p for ResNet-18 and +1%p for the ViT model | Helped |
| Use ILCs instead of last-layer retraining | Zero-shot classifier · ImageNet-Cue-Conflict and ImageNet-Silhouette | average accuracy: +2 percentage points | Helped |
| Use ILCs instead of last-layer retraining | Zero-shot classifier · ImageNet-A and ImageNet-R | average relative accuracy: +0.91 and +0.34 percentage points | Helped |
Surprising findings
- The penultimate layer is often outperformed by the pen-penultimate layer, and ViT shows little or negative gain from intermediate layers in some few-shot settings.
- Simply retraining the last layer on ID data already improves OOD performance over the base ERM model (e.g., Waterbirds 74.9% to 79.4%).
Limitations
- Layer selection relies on an OOD validation set, so zero-shot results depend on held-out OOD data for model selection.
- Gains are dataset- and architecture-dependent; ViT shows smaller or negative improvements in some few-shot settings.
- Sensitivity analysis uses a global distance metric and a per-feature TVD, which are heuristic proxies for invariance.
Experimental setup
Datasets: CMNIST CelebA Waterbirds MultiCelebA CIFAR-10 CIFAR-100 CIFAR-10C CIFAR-100C ImageNet-1K ImageNet-A ImageNet-R ImageNet-Cue-Conflict
Benchmarks: CMNIST CelebA Waterbirds MultiCelebA CIFAR-10C CIFAR-100C ImageNet-A ImageNet-R ImageNet-Cue-Conflict ImageNet-Silhouette
Metrics: accuracy worst-group accuracy
Baselines: Last-layer retraining (DFR) JTT Correct-N-Contrast MOO Base ERM model
Builds on: last-layer retraining (DFR) linear probing
Closest papers by meaning
- Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection (NeurIPS 2025 · similarity 0.89)
- Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness (ICLR 2024 · similarity 0.86)
- What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.86)
- ProHOC: Probabilistic Hierarchical Out-of-Distribution Classification via Multi-Depth Networks (CVPR 2025 · similarity 0.85)
- Feature Contamination: Neural Networks Learn Uncorrelated Features and Fail to Generalize (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.