Research map / Label-efficient and robust learning / OOD and anomaly detection

MaNo: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts

NeurIPS 2024 · Label-efficient and robust learning › OOD and anomaly detection · approach: Uncertainty and misclassification detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MaNo, a training-free unsupervised accuracy estimator that takes the entry-wise L_p norm (p=4) of a normalized logit matrix, with a criterion-switched Taylor/softmax normalization (softrun) to handle prediction bias under synthetic, natural and subpopulation shifts.

Problem

Estimating a pre-trained classifier's test accuracy on out-of-distribution data without labels; existing logit-based methods (ConfScore, Nuclear norm) are prone to overconfidence and prediction bias, especially under natural shift.

Approach

Grounds logit-based accuracy estimation in the low-density separation assumption, normalizes logits with a data-dependent choice between softmax and a second-order Taylor truncation of exp (softrun), and aggregates the normalized prediction matrix with an entry-wise L_p norm, which is shown to equal a function of the average Tsallis entropy.

Key results

• Under natural shift, MaNo reaches average R^2 0.913 on Office-Home and 0.949 on DomainNet, versus 0.855 and 0.899 for the next best method (Nuclear), and 0.946 vs 0.891 on PACS (R^2).
• Under subpopulation shift, MaNo achieves R^2 above 0.960 on average, with Spearman rho of 0.992–0.996 across BREEDS datasets.
• Adding softrun to Nuclear raises its R^2 on poorly calibrated Office-Home from 0.692 to 0.826 and on PACS from 0.609 to 0.851.

Measured effects reported in the paper

Surprising findings

• Softmax normalization is best on well-calibrated Office-Home while second-order Taylor normalization is best on miscalibrated PACS, so no single normalization works across all calibration scenarios.
• ConfScore is particularly strong with ConvNeXt architectures, while MaNo works best with ResNets and ViT.

Limitations

• Depends on the hyperparameter eta in the calibration-based selection criterion, fixed at 5 across experiments.
• Assumes low-density separation; logits are uninformative when the model is confidently wrong (high confidence, high bias).

Experimental setup

Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet PACS Office-Home DomainNet RR1-WILDS BREEDS (Living-17, Nonliving-26, Entity-13, Entity-30)

Benchmarks: CIFAR-10C CIFAR-100C ImageNet-C TinyImageNet-C PACS Office-Home DomainNet RR1-WILDS BREEDS Living-17 BREEDS Nonliving-26 BREEDS Entity-13 BREEDS Entity-30

Metrics: R^2 Spearman rho MAE

Baselines: Rotation ConfScore Entropy AgreeScore ATC Fréchet (AutoEval) Dispersion ProjNorm MDE COT Nuclear

Builds on: Nuclear norm accuracy estimation low-density separation assumption Tsallis entropy ATC AutoEval

Closest papers by meaning

• Mitigating Neural Network Overconfidence with Logit Normalization (ICML 2022 · similarity 0.88)
• Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.83)
• Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.83)
• Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration (NeurIPS 2026 · similarity 0.83)
• Maximum Likelihood Estimation is All You Need for Well-Specified Covariate Shift (ICLR 2024 · similarity 0.83)
• Set Learning for Accurate and Calibrated Models (ICLR 2024 · similarity 0.82)