Research map / Label-efficient and robust learning / Recognition and long-tail
Transitional Uncertainty with Layered Intermediate Predictions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TULIP, a single-pass uncertainty method that attaches internal classifiers (shallow-deep network exits) to intermediate layers of a ResNet and fuses their Gaussian-process uncertainty scores via logistic regression fitted on unlabeled in-distribution validation data.
Problem
Single-pass uncertainty estimation in neural networks requires feature preservation, but enforcing distance preservation in the output (via spectral normalization or gradient penalties) can harm information compression and generalization, especially under class imbalance.
Approach
Instead of constraining the output representation, preserve feature distances across a collection of intermediate representations (transitional feature preservation) by extracting features from intermediate layers with internal classifiers before collapse, with no explicit preservation constraint; the combination weights are fit from disagreement-based proxy labels without OOD data.
Key results
- On CIFAR10 with ResNet-50, TULIP reached 0.738 CIFAR10-C AUROC versus 0.721 for SNGP and 0.725 for the 5-member ensemble, at 1x runtime and 1.5x space.
- On CIFAR100 with ResNet-50, TULIP reached 78.437% accuracy and 0.868 CIFAR10-C AUROC, outperforming SNGP (75.083%, 0.821) and the 5-ensemble (79.475%, 0.828 AUROC).
- On ImageNet with ResNet-101, TULIP lowered ECE to 0.678 from 1.138 for the DNN and achieved 0.885 ImageNet-C AUROC versus 0.867 for the DNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use intermediate classifier uncertainty scores with fitted combination weights | TULIP on CIFAR10 · CIFAR10-C | AUROC: 0.715 ± 0.007 -> 0.738 ± 0.006 | Helped |
| Use intermediate classifier uncertainty scores with fitted combination weights | TULIP on CIFAR10 · CIFAR100-C | AUROC: 0.903 ± 0.000 -> 0.936 ± 0.003 | Helped |
| Use intermediate classifier uncertainty scores with fitted combination weights | TULIP on CIFAR10 · SVHN | AUROC: 0.976 ± 0.003 vs 0.946 ± 0.012 | Mixed |
| Use TULIP instead of DNN | CIFAR100 with ResNet-50 · CIFAR10-C | AUROC: 0.832 ± 0.003 -> 0.868 ± 0.002 | Helped |
| Use TULIP instead of DNN | CIFAR100 with ResNet-50 · CIFAR100-C | AUROC: 0.680 ± 0.004 -> 0.738 ± 0.002 | Helped |
| Use TULIP instead of DNN | CIFAR100 with ResNet-50 · SVHN | AUROC: 0.859 ± 0.013 -> 0.955 ± 0.008 | Helped |
| Add intermediate classifiers post-hoc to a pretrained network | ResNet-101 on ImageNet · ImageNet | ECE: 1.138 ± 0.005 -> 0.678 ± 0.001 | Helped |
| Add fitted combination weights instead of equal weighting (ablation) | TULIP · CIFAR10-C | AUROC: 0.716 ± 0.006 -> 0.738 ± 0.006 | Helped |
| Add fitted combination weights instead of equal weighting (ablation) | TULIP · CIFAR100-C | AUROC: 0.883 ± 0.009 -> 0.936 ± 0.003 | Helped |
| Add fitted combination weights instead of equal weighting (ablation) | TULIP · SVHN | AUROC: 0.840 ± 0.023 -> 0.946 ± 0.012 | Helped |
| Add a GP layer to internal classifiers (ablation) | TULIP with switch weighting · CIFAR10-C | AUROC: 0.722 ± 0.005 -> 0.738 ± 0.006 | Helped |
| Add a GP layer to internal classifiers (ablation) | TULIP with switch weighting · CIFAR100-C | AUROC: 0.885 ± 0.008 -> 0.936 ± 0.003 | Helped |
Surprising findings
- Spectral normalization-based SNGP accuracy drops by nearly three percent when extending ResNet-50 to ResNet-101 on CIFAR10.
- Omitting the preservation constraint on the first layer did not compromise uncertainty quality, suggesting the first layer rarely collapses data points in practice.
Limitations
- Adding internal classifiers increases space complexity and introduces many hyperparameters.
- Shallow architectures offer fewer intermediate representations, making the approach less effective there.
- Performance is not uniformly best; the 5-member ensemble and DUN/SNGP win some individual columns, and the medical results were called matching or outperforming without clear dominance.
Experimental setup
Datasets: CIFAR10 CIFAR100 CIFAR10-C CIFAR100-C SVHN ImageNet ImageNet-C organA organC organS
Benchmarks: CIFAR10 vs CIFAR10-C/CIFAR100-C/SVHN CIFAR100 vs CIFAR10-C/CIFAR100-C/SVHN ImageNet ImageNet-C organA/organC/organS CT datasets
Metrics: AUROC classification accuracy ECE
Baselines: Ensemble of 5 Energy-Based model DUN Early Exit SNGP DUQ MC-Dropout
Builds on: shallow-deep networks SNGP DUQ deep ensembles
Closest papers by meaning
- Deep Deterministic Uncertainty: A New Simple Baseline (CVPR 2023 · similarity 0.87)
- Distribution-Free Data Uncertainty for Neural Network Regression (ICLR 2025 · similarity 0.86)
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.86)
- Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks (ICLR 2024 · similarity 0.85)
- Uncertainty Estimation for Heterophilic Graphs Through the Lens of Information Theory (ICML 2025 · similarity 0.85)
- Epistemic Uncertainty Quantification For Pre-Trained Neural Networks (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.