Research map / Architectures and efficiency / Pruning and efficient tuning

Pairwise AUC Optimization Needs Corrective Power: A Unified View

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes why direct pairwise AUC-surrogate SGD can lose "corrective power" on severe misrankings via U-statistic Hoeffding variance collapse, and explains squared-hinge, CE warm-up, and PEFT effects with CV and NLP experiments.

Problem

Direct pairwise AUC-surrogate training aligns the objective with the AUC metric, but stochastic mini-batch updates may fail to correct severe positive-negative misrankings, leaving training stuck in regions with many severe errors and weak useful motion.

Approach

Introduces Corrective Power and a bad-region characterization based on weak projected mean drift along a severe-error correction direction; shows the pairwise mini-batch gradient is a two-sample U-statistic whose first-order noise can vanish, shifting escape-time scale from B to B^2. Uses surrogate weight geometry (squared-hinge) and starting-region interventions (CE warm-up, PEFT) to explain empi

Key results

• Squared-hinge gives the highest AUC in most displayed DenseNet121 and Qwen2.5-1.5B-Instruct settings (e.g., ANLI R1 E/N direct 86.85 vs CE 86.01).
• Direct hinge can collapse badly on NLP tasks (e.g., ANLI R3 C/N direct 52.02±2.09) while CE warm-up recovers it to 70.37±1.65.
• In a 1D escape simulation, 76% of runs cross the barrier under first-order B^-1 noise versus 3% under collapsed B^-2 noise.

Surprising findings

• Direct hinge training can keep the logged-batch misranking ratio high for many updates despite being aligned with AUC.
• Under PEFT on Qwen2.5-1.5B-Instruct, direct squared-hinge is often already close to its CE-warm-up counterpart, reducing the need for warm-up.

Limitations

• Theoretical results are local and conditional on bad-region and trajectory-containment assumptions.
• Warm-up and PEFT effects are not uniform across losses or models; gains are variable.
• Full pairwise AUC is used as the clean setting; partial AUC and auxiliary-state methods are not analyzed.

Experimental setup

Datasets: ImageNet-LT Stanford Dogs FGVC-Aircraft APTOS diabetic-retinopathy DeepWeeds D-KAP apple leaf disease Rice Leaf Diseases HAM10000 GoEmotions ANLI SNLI WANLI

Benchmarks: ImageNet-LT Dogs Dog Breeds CRJ700/CRJ900 APTOS Sev./Mod. DeepWeeds P/S Apple Disease Rice Disease HAM10000 M/N GoEmotions Annoyance GoEmotions Fear ANLI R1 ANLI R3

Metrics: AUC AUC gain from warm-up misranking ratio

Baselines: cross-entropy square loss hinge loss exponential loss logistic loss

Builds on: LoRA Deep AUC maximization Hoeffding decomposition

Closest papers by meaning

• Stability-based Generalization Analysis of Randomized Coordinate Descent for Pairwise Learning (AAAI 2025 · similarity 0.85)
• A Closer Look at AUROC and AUPRC under Class Imbalance (NeurIPS 2024 · similarity 0.85)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations (ICLR 2025 · similarity 0.85)
• Error Analysis Affected by Heavy-Tailed Gradients for Non-Convex Pairwise Stochastic Gradient Descent (AAAI 2025 · similarity 0.84)
• MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.84)
• Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques (NeurIPS 2024 · similarity 0.84)