Research map / Label-efficient and robust learning / Recognition and long-tail
Label Alignment Regularization for Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes label alignment regularization (LAR), which removes source-domain implicit label-alignment regularization of least squares and adds a target-domain regularizer on the unlabeled data's top singular vectors, adapting linear classifiers without an optimal-joint-risk assumption.
Problem
Unsupervised domain adaptation fails when label distributions shift (high optimal joint error), where representation-matching methods like DANN break down.
Approach
Exploit the label alignment property: regularize the linear classifier's predictions on the unlabeled target data toward its top right singular vectors (and remove the implicit source regularization of least squares), instead of regularizing representations.
Key results
- On MNIST-USPS (binary, averaged over 45 tasks), LAR reaches 82.71% at 0.1 subsampling vs 47.27% for DANN and 75.41% for linear no-adaptation.
- On XED cross-lingual sentiment, LAR achieves 58.55 average F1 vs 52.56 for IWDAN and 48.88 for Adv-Refine (MSE), best on 7 of 9 pairs.
- Theory shows the LAR solution lies in the span of the top-k target right singular vectors and is independent of lambda under stated assumptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply target-domain label alignment regularization | Linear model · MNIST-USPS M→U | accuracy: 83.84 -> 88.96 | Helped |
| Apply target-domain label alignment regularization | Linear model · MNIST-USPS 0.3→U, 0.2→U, and 0.1→U | accuracy: 80.99 -> 86.99; 79.47 -> 84.84; 75.41 -> 82.71 | Helped |
| Replace target-domain label alignment regularization with DANN | MNIST-USPS adaptation · MNIST-USPS U→M and M→U | accuracy: 77.85 -> 83.93; 84.88 -> 86.69 | Mixed |
| Use DANN instead of no adaptation | MNIST-USPS adaptation · MNIST-USPS 0.3→U, 0.2→U, and 0.1→U | accuracy: 83.36 -> 78.05; 72.84 -> 64.2; 53.58 -> 47.27 | Hurt |
| Apply target-domain label alignment regularization | Linear model · XED cross-lingual sentiment, average over 9 language pairs | F1: 52.56 (0.84) -> 58.55 (0.17) | Helped |
| Apply multiclass target-domain label alignment regularization | Linear model · MNIST-USPS multiclass benchmark | accuracy: 37.53 -> 42.47; 54.41 -> 63.90; 48.71 -> 54.69; 46.21 -> 51.70; 41.54 -> 47.49 | Helped |
| Remove the source-domain implicit label alignment regularization term (ablation) | Label Alignment Regression · Synthetic distribution shift | target accuracy | Helped |
| Increase training from 2k to 20k epochs (ablation) | Label Alignment Regression · Synthetic distribution shift | target accuracy sensitivity to λ: 2k; 20k | Helped |
| Vary λ in Label Alignment Regression (ablation) | Label Alignment Regression · MNIST-USPS multiclass benchmark | accuracy | No clear effect |
Surprising findings
- DANN performs worse than a regular neural network without adaptation under severe label subsampling, while its domain discriminator accuracy stays near 50%.
- Domain-adversarial methods (CDAN, f-DAL) often fail to beat no-adaptation baselines on the XED tasks.
Limitations
- Requires a small labeled target validation set (100 points) for hyperparameter selection; no fully unsupervised selection strategy.
- Multi-class extension is preliminary and needs per-output hyperparameter tuning; theory requires assumptions such as invertibility and k = k-tilde.
- Label alignment property is only verified empirically on a limited set of tasks and is not rigorously characterized.
Experimental setup
Datasets: MNIST USPS XED UCI CT Scan Million Song Year UCI Bike Sharing CIFAR-10 CIFAR-100 STL-10 AG News
Benchmarks: MNIST-USPS XED cross-lingual sentiment
Metrics: accuracy F1 score standard error
Baselines: No Adaptation (NN) DANN No Adaptation (Linear) Adversarial-Refine CDAN f-DAL IWDAN IWCDAN l2 regularization
Builds on: domain-adversarial neural networks (DANN) label alignment property Adversarial-Refine CDAN f-DAL
Closest papers by meaning
- Learning Label Shift Correction for Test-Agnostic Long-Tailed Recognition (ICML 2024 · similarity 0.81)
- Heterogeneous Label Shift: Theory and Algorithm (ICML 2025 · similarity 0.80)
- Navigating Image Restoration with VAR's Distribution Alignment Prior (CVPR 2025 · similarity 0.80)
- Addressing Label Shift in Distributed Learning via Entropy Regularization (ICLR 2025 · similarity 0.80)
- Pairwise Alignment Improves Graph Domain Adaptation (ICML 2024 · similarity 0.79)
- Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillation (NeurIPS 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.