Research map / Architectures and efficiency / RNNs and normalization
Sketched Lanczos uncertainty score: a low-memory summary of the Fisher information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sketched Lanczos Uncertainty (SLU), which combines Lanczos with SRFT sketching to approximate the top eigenvectors of the GGN/Fisher with memory growing only logarithmically in parameter count, enabling a higher-rank local-ensemble OoD score under a fixed memory budget.
Problem
Uncertainty quantification via low-rank GGN/Fisher approximations needs k·p memory, which limits the approximation rank for large neural networks under tight memory budgets.
Approach
Sketch each Lanczos vector on the fly with an SRFT, orthogonalize the sketches post-hoc (sketching and orthogonalization approximately commute), so memory is s·k with s only log-dependent on p; the saved memory allows a higher rank k than baselines at the same budget.
Key results
- Under a 3p memory budget, SLU achieves AUROC 0.94 on MNIST vs FashionMNIST-style OoD for MLP and 0.94 for LeNet MNIST-OoD vs baselines around 0.80-0.84 (LLA, SCOD) and 0.92 for Deep Ensemble.
- On VisualAttentionNet (CelebA, 4M params), SLU reaches 0.95 AUROC on Food-101 OoD versus 0.94 for SCOD and 0.93 for LLA.
- On SWIN ImageNet (200M params), SLU has the best AUROC on 7 of 8 OoD classes, with mean 0.72 on hold-out classes versus 0.67 for LLA, while SCOD and Deep Ensemble are not applicable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Sketched Lanczos (SRFT sketch, higher rank under same memory) vs low-rank Laplace/local ensemble baselines at budget 3p | Uncertainty score on ResNet/VisualAttentionNet/SWIN models · OoD detection, multiple ID-OoD pairs | AUROC | Mixed |
| Deep Ensemble vs memory-budgeted single-model scores | Deep Ensemble baseline · Growing model size (MLP to ResNet) | AUROC | Hurt |
| Diagonal Laplace (1p memory) instead of low-rank scores | MLP on MNIST · MNIST vs FashionMNIST OoD | AUROC: 0.94 vs 0.26 for SLU | Helped |
| Preconditioning (hi-memory Lanczos warm start before sketched Lanczos) | Sketched Lanczos · Numerical stability | qualitative | No clear effect |
Surprising findings
- Diagonal Laplace (LLA-D) outperforms SLU on MLP MNIST OoD tasks (e.g. 0.94 vs 0.26 on FashionMNIST OoD), showing the low-rank advantage is not uniform.
- Deep Ensemble performs well on small architectures but deteriorates for larger parameter counts.
Limitations
- The method does not allow sampling from the approximate posterior, only computing predictive variance.
- Query cost scales linearly with output dimension t, which can slow inference for generative models.
- Performance is not uniformly better: diagonal Laplace wins on some MLP settings and SWAG on some CIFAR-10 corruptions; SWAG is sensitive to hyperparameters.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10 CelebA ImageNet
Benchmarks: MNIST FashionMNIST KMNIST CIFAR-10 CIFAR-100 SVHN CelebA Food-101 ImageNet
Metrics: AUROC
Baselines: LLA LLA-D Local Ensemble Local Ensemble Hessian variant SCOD SWAG Deep Ensemble
Builds on: Local Ensembles Linearized Laplace approximation SCOD Lanczos algorithm Subsampled Randomized Fourier Transform
Closest papers by meaning
- Uncertainty Quantification with the Empirical Neural Tangent Kernel (NeurIPS 2025 · similarity 0.86)
- A Unified Uncertainty Representation for Graph Neural Networks via Doubly-Spectral Stochastic Expansion (NeurIPS 2026 · similarity 0.85)
- Epistemic Uncertainty Quantification For Pre-Trained Neural Networks (CVPR 2024 · similarity 0.84)
- Trade-Offs of Diagonal Fisher Information Matrix Estimators (NeurIPS 2024 · similarity 0.84)
- Uncertainty Modeling in Graph Neural Networks via Stochastic Differential Equations (ICLR 2025 · similarity 0.84)
- Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.