Research map / Label-efficient and robust learning / Recognition and long-tail
Contrastive dimension reduction: when and how?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a contrastive bootstrap hypothesis test and a contrastive dimension estimator (CDE) based on principal angles between foreground and background subspaces to decide whether contrastive dimension reduction is warranted and how many foreground-unique dimensions exist.
Problem
Contrastive dimension reduction methods are used without a way to test whether foreground-unique information exists, and the reduced dimension must be chosen as a tuning parameter without a principled estimator.
Approach
Define the contrastive subspace as the projection of the foreground subspace onto the orthogonal complement of the background subspace; its dimension equals the count of nonzero principal angles (plus max(dx-dy,0)). Test H0 (foreground subspace contained in background) via a contrastive bootstrap, and estimate the contrastive dimension by counting principal-angle cosines below a threshold.
Key results
- On Corrupted MNIST the hypothesis test gives p=0.008 and CDE estimates a contrastive dimension of 5 (threshold 0.9), with logistic-regression accuracy plateauing near 0.945 at 5 CPCA components.
- On mHealth the test rejects H0 (p<0.001) and CDE estimates contrastive dimension 1, consistent with prior CPCA analysis.
- In simulation 1 (dxy=0) the test is conservative, with empirical type I error below nominal levels; simulation 2 (dxy=6) gives p=0.029 and correct estimate 6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CDE to foreground and background with true dxy=0 | CDE · Simulation 1 | estimated contrastive dimension: 0 -> 0.44 | Helped |
| apply CDE where true dxy=6 | CDE · Simulation 2 | estimated contrastive dimension: 6 -> 6 | Helped |
| apply CDE to disk images with a foreground line | CDE · Disk + Line | estimated contrastive dimension: 1 -> 1 | Helped |
| apply CDE to MNIST images with a foreground line | CDE · MNIST + Line | estimated contrastive dimension: 1 -> 1 | Helped |
| apply contrastive bootstrap test under the null setting | Contrastive bootstrap test · Simulation 1 | type I error | Mixed |
| apply contrastive bootstrap test where dxy=6 | Contrastive bootstrap test · Simulation 2 | p-value: 0.029 | Helped |
| use 2 instead of 1 CPCA component | CPCA classification · Corrupted MNIST classification | accuracy: 0.583 -> 0.941 | Helped |
| increase CPCA dimension from 2 to 5 | CPCA classification · Corrupted MNIST classification | accuracy: 0.941 -> 0.945 | Helped |
| increase CPCA dimension from 5 to 10 | CPCA classification · Corrupted MNIST classification | accuracy: 0.945 -> 0.947 | No clear effect |
| apply CDE to Noisy MNIST | CDE · Noisy MNIST | estimated contrastive dimension: 5 | Helped |
| apply the bootstrap test to BMMC | Contrastive bootstrap test · BMMC | p-value: 0.073 | Mixed |
| apply CDE to Pathogen data | CDE · Pathogen | estimated contrastive dimension: 10 | Helped |
Surprising findings
- The Pathogen dataset yields p=0.161, suggesting linear contrastive methods may be inappropriate there, consistent with CVI outperforming linear methods.
- The hypothesis test is conservative, so it fails to reject for BMMC (p=0.073) and CelebA despite CDE estimating nonzero contrastive dimensions.
Limitations
- The hypothesis test is conservative and less powerful; a likelihood-ratio alternative is not yet solved.
- Linear subspace model; nonlinear manifold structure can make the test and estimate inconsistent.
- Results depend on the intrinsic dimension estimates and the subjective singular-value cutoff (0.9 or 0.1 tolerance).
Experimental setup
Datasets: MNIST Corrupted MNIST Mouse Protein mHealth BMMC Small Molecule ECCITE-Seq Pathogen Perturb-Seq CelebA
Benchmarks: Corrupted MNIST Mouse Protein mHealth BMMC Small Molecule ECCITE-Seq Pathogen Perturb-Seq CelebA
Metrics: p-value estimated contrastive dimension principal angle cosines type I error rate classification accuracy
Baselines: CPCA PCPCA CLVM CVAE CVI
Builds on: Contrastive PCA Probabilistic Contrastive PCA Contrastive Latent Variable Model Contrastive Variational Inference Probabilistic PCA
Closest papers by meaning
- Contrastive Functional Principal Component Analysis (AAAI 2025 · similarity 0.83)
- Dimension Reduction with Locally Adjusted Graphs (AAAI 2025 · similarity 0.82)
- Separating common from salient patterns with Contrastive Representation Learning (ICLR 2024 · similarity 0.81)
- DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation (ICML 2024 · similarity 0.81)
- Enhancing Sufficient Dimension Reduction via Hellinger Correlation (ICML 2024 · similarity 0.81)
- How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction (ICML 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.