Research map / Label-efficient and robust learning / Recognition and long-tail
Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SelMix, a selective mixup fine-tuning method that samples class pairs for feature mixup according to an estimated gain on a non-decomposable metric, to optimize objectives like min recall, H-mean, G-mean and coverage-constrained mean recall on pre-trained long-tailed models.
Problem
Optimizing non-decomposable, fairness-constrained metrics (worst-case recall, H-mean, G-mean, coverage-constrained recall) for deep networks on imbalanced data; existing empirical methods do well on mean recall but poorly on these, while theoretical methods require retraining from scratch per object
Approach
Fine-tune a pre-trained feature extractor and linear head by mixing features of class pairs (i,j) sampled from a softmax over estimated gains G_ij = directional derivative of the metric along the mixup-induced weight update; gains are computed analytically from the confusion matrix via an unconstrained reparameterization, avoiding retraining per objective.
Key results
- On CIFAR-10-LT (rho=100) semi-supervised, SelMix with FixMatch(LA) reaches 79.1 min recall vs 71.3 for the best baseline CReST+ and 55.9 for FixMatch(LA).
- On CIFAR-100-LT min head-tail recall rises from 34.6 (FixMatch LA) to 57.8 with SelMix, a 9.8-point gain over the next-best baseline, while mean recall stays at 59.8.
- On ImageNet-1k-LT, SelMix raises min head-tail recall from 29.7 (MiSLAS stage 2) to 45.1 with mean recall 52.8 vs 52.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune FixMatch (LA) with SelMix | FixMatch (LA) · CIFAR-10 LT (semi-supervised, matched distributions) | Min recall: 55.9 ± 1.9 -> 79.1 ± 0.1 | Helped |
| Fine-tune FixMatch (LA) with SelMix | FixMatch (LA) · CIFAR-100 LT (semi-supervised, matched distributions) | Min head-tail recall: 34.6 ± 0.6 -> 57.8 ± 0.5 | Helped |
| Fine-tune MiSLAS with SelMix instead of Stage 2 | MiSLAS · CIFAR-10 LT (supervised) | Min recall: 72.5 ± 0.8 -> 79.2 ± 0.7 | Helped |
| Fine-tune MiSLAS with SelMix instead of Stage 2 | MiSLAS · CIFAR-100 LT (supervised) | Min head-tail recall: 15.2 ± 1.1 -> 41.3 ± 1.4 | Helped |
| Fine-tune FixMatch (LA) with SelMix | FixMatch (LA) · ImageNet-100 LT | Min recall: 6.0 -> 24.0 | Helped |
| Fine-tune MiSLAS with SelMix instead of Stage 2 | MiSLAS · ImageNet-1k LT | Min head-tail recall: 29.7 -> 45.1 | Helped |
| Fine-tune FixMatch (LA) with SelMix | FixMatch (LA) · CIFAR-100 LT (semi-supervised, matched distributions) | Mean recall: 58.8 ± 0.1 -> 59.8 ± 0.2 | Hurt |
Surprising findings
- Pre-training with CSST on long-tailed data yields sub-optimal representations that hurt mean recall, motivating fine-tuning from a pre-trained model instead.
- A purely greedy mixup policy (infinite inverse temperature) fails in practice because the linear gain approximation breaks down over multiple SGD steps.
Limitations
- Theory relies on linear-classifier approximations and assumptions such as small feature variance and a concave, Lipschitz objective; the gain approximation is only empirically validated.
- Fine-tunes mainly the linear head, with the backbone at a lower learning rate, and gains are derived for linear classifiers.
- Requires a labeled validation set to estimate class centroids and gains, and periodic pseudo-label updates.
Experimental setup
Datasets: CIFAR-10-LT CIFAR-100-LT ImageNet-100-LT ImageNet-1k-LT STL-10
Benchmarks: CIFAR-10-LT CIFAR-100-LT ImageNet-100-LT ImageNet-1k-LT STL-10
Metrics: mean recall min recall H-mean G-mean min coverage min head-tail recall
Baselines: DARP CReST CReST+ ABC CoSSL DASO CSST MiSLAS
Builds on: FixMatch Manifold Mixup CSST MiSLAS
Closest papers by meaning
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.84)
- BEM: Balanced and Entropy-based Mix for Long-Tailed Semi-Supervised Learning (CVPR 2024 · similarity 0.84)
- Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup (ICML 2024 · similarity 0.84)
- Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning (ICML 2024 · similarity 0.84)
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.84)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.