Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Shuffled Deep Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Shuffled Deep Regression (SDR), a sparse stochastic EM algorithm (SSEM) for training neural-network regressors from shuffled input-output data without known correspondences.
Problem
Learning regression models from shuffled data where the correspondence between input features and target outputs is unknown; prior methods are largely limited to linear or simple models.
Approach
Combines sparse EM (restricting the posterior over permutations to a small candidate set built from a sort-based proposition) with stochastic gradient updates (Adam) so that a one-hidden-layer MLP can be trained on shuffled data, and links the procedure to IRLS for linear models.
Key results
- SDR outperforms the linear shuffled-regression baseline SLR in almost all settings on UCI benchmarks (e.g., MPG K=2 test MSE 12.31 vs SLR 9.53 is an exception; Boston K=2 SDR 33.68 vs SLR 11.68 is worse, so results are mixed by setting)
- SDR is comparable to the paired-data deep regression oracle DR in several settings, e.g., Abalone K=2 (4.98 vs 4.98) and MPG-1D K=2 to 16.
- SDR overtakes SLR as the number of shuffled sets n grows (Fig. 3b,c), and SLR-based initialization of candidate permutations speeds convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SDR instead of SLR | shuffled regression · MPG-1D, MPG, Abalone, Boston, Concrete (K=2, 4, 8, 16) | test MSE | Helped |
| Initialize permutation candidates using SLR instead of random initialization (ablation) | SDR · Abalone (K=8) | convergence speed | Helped |
| Vary the number of hidden units (ablation) | SDR · Abalone (K=8) | test MSE | Mixed |
| Vary the learning rate (ablation) | SDR · Abalone (K=8) | test MSE | No clear effect |
Surprising findings
- SDR is often worse than the linear SLR baseline at small data sizes and larger K (e.g., MPG K=16 DR-style error 24.60 vs SLR 21.40 and Boston K=16 SDR 50.86 vs SLR 48.57), indicating deep models need more data.
Limitations
- Theoretical analysis of how data size n, set size K and input dimension d affect performance is left for future work.
- Deep model needs large amounts of shuffled data; performance is worse than linear SLR when n is small.
- Evaluated only on small UCI tabular datasets with a single one-hidden-layer MLP.
Experimental setup
Datasets: auto-MPG (MPG) Abalone Boston housing Concrete compressive strength MPG-1D (horsepower only)
Benchmarks: MPG-1D MPG Abalone Boston Concrete
Metrics: test mean squared error
Baselines: SLR (sparse-EM linear shuffled regression, extension of Abid and Zou 2018) LR (oracle linear regression on paired data) DR (oracle deep regression on paired data)
Builds on: Expectation-Maximization sparse EM (Neal and Hinton 1998) online EM Abid and Zou 2018 EM for shuffled linear regression Adam
Closest papers by meaning
- Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness (ICML 2025 · similarity 0.82)
- On the Last-Iterate Convergence of Shuffling Gradient Methods (ICML 2024 · similarity 0.80)
- Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift (ICML 2024 · similarity 0.80)
- Tighter Convergence Bounds for Shuffled SGD via Primal-Dual Perspective (NeurIPS 2024 · similarity 0.79)
- Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks (CVPR 2024 · similarity 0.79)
- Transformers can optimally learn regression mixture models (ICLR 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.