Research map / Probabilistic and scientific modeling / Molecules and proteins
Navigating the Effect of Parametrization for Dimensionality Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that parametric neighborhood-embedding methods lose local structure versus non-parametric ones, attributes this to weak repulsion of negatives, and proposes ParamRepulsor, which uses mid-near hard negative mining and a strong repulsive loss for parametric DR.
Problem
Parametric neighborhood-embedding dimensionality reduction methods generalize to unseen data but often produce worse local structure and blurred cluster boundaries than their non-parametric counterparts.
Approach
Diagnoses the parametric gap as insufficient repulsion of negative pairs and proposes Hard Negative Mining using mid-near pairs (second-closest of 6 sampled points) as hard negatives, combined with a PaCMAP-based loss with stronger repulsive forces, trained with an MLP projector.
Key results
- ParamRepulsor achieves the highest 10-NN accuracy on 10 of 14 datasets (e.g., MNIST 0.969 vs 0.968 for P-PaCMAP and 0.830 for P-ItSNE).
- ParamRepulsor achieves the highest SVM accuracy on 9 of 14 datasets, e.g., COIL-100 0.862 vs 0.825 for P-PaCMAP.
- ParamPaCMAP is highest on 5 of 14 datasets for centroid distance correlation and ParamRepulsor on 4, showing global structure is retained.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ParamRepulsor hard-negative mining and its repulsive loss to parametric PaCMAP | Parametric PaCMAP · MNIST, F-MNIST, COIL-20, COIL-100, Kang, Muraro, Stuart, 10-NN accuracy | 10-NN accuracy: 0.968 -> 0.969; 0.744 -> 0.778; 0.853 -> 0.887; 0.896 -> 0.928; 0.960 -> 0.961; 0.961 -> 0.962; 0.855 -> 0.856 | Helped |
| Add ParamRepulsor hard-negative mining and its repulsive loss to parametric PaCMAP | Parametric PaCMAP · USPS, 20NG, Kazer, Circle, Mammoth, 10-NN accuracy | 10-NN accuracy: 0.960 -> 0.957; 0.437 -> 0.460; 0.940 -> 0.939; 0.904 -> 0.895; 0.915 -> 0.938 | Mixed |
| Use NEG-style rather than NCE/InfoNCE-style loss | Parametric neighborhood embedding methods · MNIST | local structure preservation | Helped |
| Increase projector depth beyond three hidden layers (ablation) | P-UMAP and P-PaCMAP · MNIST | 10-NN accuracy | No clear effect |
| Increase projector depth beyond three hidden layers (ablation) | P-ItSNE · MNIST | 10-NN accuracy | No clear effect |
| Vary the number of nearest neighbors used to construct the graph (ablation) | Parametric DR methods · MNIST | embedding quality | No clear effect |
| Increase nearest neighbors to 60 (ablation) | P-ItSNE · MNIST | embedding quality | Hurt |
| Parameterize neighborhood embedding methods rather than optimize a non-parametric embedding | Neighborhood embedding methods · local structure preservation | local structure preservation | Hurt |
| Use mid-near pairs as hard negatives and apply repulsive forces to them | ParamRepulsor · MNIST | local structure preservation | Helped |
| Use mid-near rather than uniformly sampled pairs for negative sampling | ParamRepulsor · datasets with more than 1330 points | false-negative probability: O(\frac{1}{n^2}); \frac{10}{n} | Helped |
Surprising findings
- Adding MLP layers beyond three yields diminishing returns, and changing the number of nearest neighbors has minimal effect on parametric embeddings.
- Mid-near hard negative sampling yields fewer false negatives than uniform sampling only when the dataset has more than about 1330 points.
Limitations
- ParamRepulsor is slower than Parametric Info-NC-t-SNE.
- Choosing the optimal projector architecture and evaluation metrics remain open questions.
- Parametric methods still fall short of non-parametric visual quality on MNIST according to the authors.
Experimental setup
Datasets: MNIST Fashion-MNIST USPS COIL-20 COIL-100 20 Newsgroups Kang scRNA-seq Kazer scRNA-seq Muraro scRNA-seq Stuart scRNA-seq Circle Mammoth
Benchmarks: MNIST Fashion-MNIST USPS COIL-20 COIL-100 20NG Kang Kazer Muraro Stuart Circle Mammoth
Metrics: 10-NN accuracy SVM accuracy 30-NN preservation ratio triplet preservation centroid distance Spearman correlation
Baselines: Parametric UMAP Parametric Info-NC-t-SNE Parametric Neg-t-SNE Parametric NCVis GeoAE ParamPaCMAP
Builds on: PaCMAP UMAP t-SNE NCVis contrastive learning
Closest papers by meaning
- Neuc-MDS: Non-Euclidean Multidimensional Scaling Through Bilinear Forms (NeurIPS 2024 · similarity 0.85)
- Enhancing Sufficient Dimension Reduction via Hellinger Correlation (ICML 2024 · similarity 0.84)
- Finsler Multi-Dimensional Scaling: Manifold Learning for Asymmetric Dimensionality Reduction and Embedding (CVPR 2025 · similarity 0.83)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.83)
- LDReg: Local Dimensionality Regularized Self-Supervised Learning (ICLR 2024 · similarity 0.83)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.