Research map / Label-efficient and robust learning / Recognition and long-tail
Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes nuisance-aware prediction sets (NAPS) that estimate classifier ROC curves across the nuisance parameter space via monotone regression, yielding valid set-valued classifiers under generalized label shift in likelihood-free inference.
Problem
Classifying events with valid uncertainty quantification when a simulator defines the data-generating process, but the joint distribution of labels and latent nuisance parameters differs between training and target data (generalized label shift).
Approach
Casts classification as composite hypothesis testing under nuisance parameters, estimates rejection probabilities W(C; y, nu) across the whole nuisance space using monotone regression on augmented simulated calibration data, and derives cutoffs invariant to GLS; optionally tightens cutoffs using (1-gamma) confidence sets for nuisance parameters.
Key results
- On the RNA-Seq experiment, NAPS (gamma=0) achieves valid coverage across all four batch protocols, whereas standard, class-conditional and APS prediction sets undercover for at least two protocols.
- In the synthetic example under GLS, standard prediction sets undercover for all alpha levels while NAPS remains valid; constraining nuisance parameters (gamma>0) increases power.
- On the cosmic-ray experiment, NAPS with gamma>0 achieves uniformly better classification metrics than the standard Bayes classifier at all confidence levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NAPS instead of standard prediction sets under generalized label shift | Synthetic classification · Synthetic example with GLS | coverage validity | Helped |
| Constrain nuisance parameters with a confidence set (γ > 0) rather than use unrestricted NAPS (γ = 0) (ablation) | NAPS · Synthetic example with GLS | power | Helped |
| Use NAPS instead of marginal-coverage, class-conditional, or APS baselines | RNA-Seq cell-type classification · RNA-Seq, four batch protocols | coverage validity | Helped |
| Use APS at very high confidence levels | RNA-Seq prediction sets · RNA-Seq | prediction-set informativeness | Hurt |
| Constrain nuisance parameters with confidence sets (γ > 0) rather than use NAPS (γ = 0) (ablation) | NAPS · Cosmic Rays | classification performance | Helped |
| Use NAPS (γ = 0) instead of the standard Bayes classifier | Cosmic-ray classification · Cosmic Rays, lower-energy bin | classification performance | Hurt |
| Use NAPS (γ = 0) instead of the standard Bayes classifier | Cosmic-ray classification · Cosmic Rays | true positives and false negatives | Helped |
| Use NAPS (γ > 0) rather than a point prediction of nuisance parameters (ablation) | NAPS · Generalized label shift | uncertainty validity | Helped |
| Set γ > 0 rather than γ = 0 (ablation) | NAPS · Generalized label shift | power | Mixed |
| Use NAPS rather than standard prediction sets when there is no shift | Classification with nuisance parameters · No GLS | power | Hurt |
Surprising findings
- Standard prediction sets and point-estimate-based hybrid cutoffs lose marginal validity under GLS, and conformal APS yields uninformative sets containing both labels at high confidence.
- Setting gamma>0 is not guaranteed to increase power relative to gamma=0 and can decrease it when nuisance confidence sets are large or skewed.
Limitations
- Under GLS-free settings NAPS has reduced power compared with standard prediction sets
- gamma>0 cutoffs must be computed per test point, which is expensive for high-dimensional nuisance parameters
- Multiclass extension via one-vs-rest is non-trivial; NAPS may produce empty prediction sets
Experimental setup
Datasets: scDesign3 simulated scRNA-seq data (from PBMC Systematic Comparative Analysis reference) CORSIKA simulated cosmic-ray showers synthetic 1D example
Benchmarks: synthetic 1D example scDesign3 RNA-Seq simulation CORSIKA cosmic-ray simulation
Metrics: coverage power true positive rate false positive rate precision false discovery rate PIT/P-P plot diagnostics
Baselines: standard prediction sets class-conditional prediction sets conformal adaptive prediction sets (APS) Bayes classifier
Builds on: Bayes Frequentist Factor (BFF) label-conditional conformal prediction adaptive prediction sets CatBoost
Closest papers by meaning
- Graph–Smoothed Bayesian Black-Box Shift Estimator and Its Information Geometry (NeurIPS 2025 · similarity 0.85)
- PAC Prediction Sets Under Label Shift (ICLR 2024 · similarity 0.84)
- Quantifying Uncertainty in the Presence of Distribution Shifts (NeurIPS 2025 · similarity 0.84)
- Not all distributional shifts are equal: Fine-grained robust conformal inference (ICML 2024 · similarity 0.83)
- Bridging Multicalibration and Out-of-distribution Generalization Beyond Covariate Shift (NeurIPS 2024 · similarity 0.83)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.