Research map / Architectures and efficiency / Pruning and efficient tuning

Rank-Constrained Adaptation for Reliable Real-World Performance

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MARLA, a parameter-efficient group-robustness method that uses ERM misclassification-weighted covariance to pick a low-dimensional subspace and learns a rank-restricted logit correction, evaluated without subgroup labels across seven datasets.

Problem

Group robustness when relevant subgroups are unlabeled, partially known, or unknown in advance; existing group-robust methods typically rely on group annotations for training, validation, or model selection.

Approach

Weight adaptation examples by ERM true-label misclassification scores, compute a weighted covariance of penultimate features, take its top-k eigenvectors as a subspace, and learn only a rank-k additive logit correction within it (initialized at zero so it starts from ERM).

Key results

• Without group information, MARLA reports the highest WGA on five of seven datasets, e.g., 78.2% WGA on MIMIC-IV vs 72.1% for ERM and 82.8% on CelebA.
• Under partial subgroup knowledge, MARLA achieves the highest WGA in six of nine settings (e.g., 92.6% on GRACE age, 79.1% on MIMIC-IV gender).
• In the low-rank vs full-rank ablation, the rank-limited MARLA outperforms full-rank classifier fine-tuning on all five datasets (e.g., CheXpert WGA 75.3 vs 56.4).

Surprising findings

• Several group-aware methods (e.g., AFR, JTT, CVaRDRO) underperform ERM substantially on some datasets when no subgroup supervision is available.
• Proposition shows that optimizing Group DRO over known groups can increase risk on an omitted group relative to ERM, motivating partial-knowledge evaluation.

Limitations

• Relies on the base model's true-label probability to select the subspace, so miscalibration, label noise, or poisoning can cause mismatches with true subgroup harm.
• Assumes failure modes are represented in adaptation data and concentrated in a low-dimensional subspace; cannot recover features absent from the representation.
• Correction applied only to the classifier head; does not improve pretrained embeddings for transfer.

Experimental setup

Datasets: GRACE MIMIC-IV Waterbirds CelebA MultiNLI CivilComments CheXpert

Benchmarks: GRACE MIMIC-IV Waterbirds CelebA MultiNLI CivilComments CheXpert

Metrics: worst-group accuracy average accuracy unknown-group accuracy

Baselines: ERM Group DRO JTT AFR DFR CRT ReWeightCRT CVaRDRO IRM LISA DPE CnC

Builds on: Just Train Twice Automatic Feature Reweighting Deep Feature Reweighting Group DRO LoRA

Closest papers by meaning

• ExMap: Leveraging Explainability Heatmaps for Unsupervised Group Robustness to Spurious Correlations (CVPR 2024 · similarity 0.85)
• The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.85)
• Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions (ICLR 2025 · similarity 0.85)
• Improving Subgroup Robustness via Data Selection (NeurIPS 2024 · similarity 0.84)
• IW-GAE: Importance weighted group accuracy estimation for improved calibration and model selection in unsupervised domain adaptation (ICML 2024 · similarity 0.84)
• RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts (ICML 2025 · similarity 0.84)