Research map / Label-efficient and robust learning / Recognition and long-tail
Deep Regression Representation Learning with Topology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PH-Reg, a regression-specific regularizer that matches the intrinsic dimension and topology of the feature space to the target space, motivated by information-bottleneck connections.
Problem
Representation learning work focuses on classification and neglects regression; regression representations should be continuous and ordinal, so their topology (intrinsic dimension and shape) relative to the target space and its effect on generalization under the Information Bottleneck principle are
Approach
Links IB tradeoff to conditional entropies H(Y|Z) and H(Z|Y), shows H(Z|Y) bounds generalization error and is bounded by feature-space intrinsic dimension, and proposes matching the feature space's intrinsic dimension and topology (via persistent homology on minimum spanning tree edge lengths) to the target space.
Key results
- On synthetic coordinate prediction, combining L_d and L_t reduces MSE on Circle from 0.154 (baseline) to 0.013 and on Swiss Roll from 2.99 to 0.61.
- On AgeDB, PH-Reg (L_d + L_t) lowers MAE from 7.80 to 7.32 overall.
- On NYU-Depth-v2, PH-Reg improves delta1 from 0.792 to 0.807 and reduces REL from 0.153 to 0.144; OE remains best on delta1 and REL (0.811 and 0.143).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add target-aware intrinsic-dimension regularization and topology regularization | 2-layer MLP · Synthetic coordinate prediction: Swiss Roll, Mammoth, Torus, and Circle | MSE: 2.99 ± 0.43 -> 0.61 ± 0.17 (Swiss Roll); 211 ± 55 -> 49 ± 27 (Mammoth); 3.01 ± 0.11 -> 0.61 ± 0.05 (Torus); 0.154 ± 0.00 | Helped |
| Use target-aware intrinsic-dimension regularization (ablation) | 2-layer MLP · Synthetic coordinate prediction: Swiss Roll, Mammoth, Torus, and Circle | MSE: 2.99 ± 0.43 -> 0.66 ± 0.08; 211 ± 55 -> 89 ± 66; 3.01 ± 0.11 -> 0.62 ± 0.12; 0.154 ± 0.006 -> 0.090 ± 0.019 | Helped |
| Use topology regularization (ablation) | 2-layer MLP · Synthetic coordinate prediction | MSE: 2.99 ± 0.43 -> 1.83 ± 0.70 (Swiss Roll); 211 ± 55 -> 80 ± 61 (Mammoth); 3.01 ± 0.11 -> 0.95 ± 0.05 (Torus); 0.154 ± 0.00 | Mixed |
| Replace target-aware intrinsic-dimension regularization with naive intrinsic-dimension minimization (ablation) | 2-layer MLP · Synthetic coordinate prediction | MSE: 2.99 ± 0.43 -> 2.74 ± 0.85 (Swiss Roll); 211 ± 55 -> 141 ± 104 (Mammoth); 3.01 ± 0.11 -> 1.13 ± 0.06 (Torus); 0.154 ± 0. | Hurt |
| Add target-aware intrinsic-dimension regularization and topology regularization | Age estimation model · AgeDB ALL | MAE: 7.80 ± 0.12 -> 7.32 ± 0.09 | Helped |
| Add target-aware intrinsic-dimension regularization and topology regularization | Super-resolution model · Urban100 | PSNR (dB): 26.083 -> 26.179 | Helped |
| Add target-aware intrinsic-dimension regularization and topology regularization | Depth estimation model · NYU-Depth-v2 | δ1: 0.792 -> 0.807 | Helped |
| Add target-aware intrinsic-dimension regularization and topology regularization | Depth estimation model · NYU-Depth-v2 | REL: 0.153 -> 0.144 | Helped |
| Add target-aware intrinsic-dimension regularization and topology regularization | Depth estimation model · NYU-Depth-v2 | RMS: 0.512 -> 0.481 | Helped |
| Add target-aware intrinsic-dimension regularization and topology regularization | Depth estimation model · NYU-Depth-v2 | δ1: 0.811 -> 0.807 | Hurt |
| Vary λ_t upward through values at or below 10 (ablation) | PH-Reg · Swiss Roll coordinate prediction | MSE | Helped |
| Set λ_t to 1000 (ablation) | PH-Reg · Swiss Roll coordinate prediction | MSE | Hurt |
Surprising findings
- Naively minimizing the intrinsic dimension (Birdal's regularizer alone) can push feature dimension below the target dimension and performs poorly on synthetic tasks.
- Information Dropout slightly hurts performance on several tasks, e.g., Mammoth MSE rises from 211 to 367.
Limitations
- Regularizer is applied to a limited number of samples per batch, so it is constrained by batch size for image-wise tasks.
- Gains on real-world tasks are modest; the authors state they do not aim for state of the art, and OE outperforms PH-Reg on some depth metrics.
- Added training time on small MLP experiments is substantial (MST computation), e.g., 8.88s to 617s.
Experimental setup
Datasets: Swiss roll torus circle mammoth (synthetic point clouds) NYU-Depth-v2 AgeDB DIV2K Set5 Set14 B100 Urban100
Benchmarks: NYU-Depth-v2 AgeDB DIV2K Set5 Set14 B100 Urban100 synthetic Swiss roll/mammoth/torus/circle coordinate prediction
Metrics: MSE MAE PSNR delta1 REL RMS log10
Baselines: Information Dropout Ordinal Entropy Birdal's regularizer (L'_d) Topology autoencoder (L_t)
Builds on: Information Bottleneck Birdal intrinsic dimension regularizer Topology autoencoder Information Dropout Ordinal Entropy
Closest papers by meaning
- Improving Deep Regression with Tightness (ICLR 2025 · similarity 0.86)
- Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms (NeurIPS 2024 · similarity 0.84)
- Cauchy-Schwarz Divergence Information Bottleneck for Regression (ICLR 2024 · similarity 0.84)
- Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samples (ICML 2024 · similarity 0.84)
- High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws (ICLR 2025 · similarity 0.84)
- Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.