Research map / Learning theory and algorithms / Prediction and decision losses
A Differentiable Rank-Based Objective for Better Feature Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes difFOCI, a differentiable parametric relaxation of the rank-based FOCI conditional-dependence coefficient, usable for feature selection, neural feature learning, spurious-correlation regularization, and fairness.
Problem
Rank-based, nonparametric conditional-dependence measures (FOCI / Chatterjee's coefficient) are non-differentiable and so cannot be used as objectives for training neural networks or for feature learning.
Approach
Replace the hard nearest-neighbor ranking in the FOCI estimator T_n with a softmax-weighted (temperature beta) soft neighbor selection over pairwise distances, making the rank-based conditional dependence differentiable with respect to learned features, and show it converges to the same limit as T as beta grows.
Key results
- On Waterbirds, ERM + NN-difFOCI reaches 85.7% worst-group test accuracy versus 60.0% for ERM, while ERM + FOCI drops to 0.0%.
- On the Toxicity/Spambase/QSAR feature selection benchmark, vec-difFOCI reaches the lowest logistic test loss on Spambase (2.56) versus 2.92 for the best baseline.
- In the fairness experiments, difFOCI features reduce sensitive-attribute test accuracy, e.g., Student data from 71.21% to 48.89%, while keeping target accuracy roughly stable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use difFOCI NN instead of FOCI for feature selection | feature selection · Spambase | logistic test loss: 3.04 -> 2.57 | Helped |
| Use difFOCI NN instead of FOCI for feature selection | feature selection · Toxicity | logistic test loss: 16.30 -> 9.61 | Helped |
| Use difFOCI NN instead of FOCI for feature selection | feature selection · QSAR | logistic test loss: 3.44 -> 2.11 | Helped |
| Use difFOCI NN instead of FOCI for feature selection | feature selection · Breast Cancer | logistic test loss: 0.62 -> 0.00 | Helped |
| Use difFOCI NN instead of FOCI for feature selection | feature selection · Religious | logistic test loss: 0.53 -> 0.56 | No clear effect |
| Add difFOCI NN regularization | ERM on Waterbirds · Waterbirds | test worst group accuracy: 60.0 -> 85.7 | Helped |
| Add difFOCI NN regularization | DRO on Waterbirds · Waterbirds | test worst group accuracy: 76.9 -> 87.2 | Helped |
| Use difFOCI+DRO instead of DRO | DRO · MetaShift | worst group accuracy: 86.2 -> 91.7 ± 0.2 | Helped |
| Use vector parameterization instead of neural-network parameterization (ablation) | difFOCI · Spambase | logistic test loss: 2.57 ± 0.19 vs 2.56 ± 0.13 | No clear effect |
| Replace NN parameterization with vec-NN parameterization (ablation) | difFOCI · not specified | performance | No clear effect |
| Add difFOCI regularization to ERM instead of using FOCI regularization | ERM on Waterbirds · Waterbirds | test worst group accuracy: 0.0 -> 85.7 | Helped |
| Add difFOCI regularization to DRO instead of using FOCI regularization | DRO on Waterbirds · Waterbirds | test worst group accuracy: 3.9 -> 87.2 | Helped |
Surprising findings
- The original FOCI performs very poorly as a deep learning regularizer, with worst-group accuracy near 0% on Waterbirds.
- The authors report that a vec-NN parameterization did not improve over the NN parameterization.
Limitations
- Softmax temperature beta and cutoff were fixed (beta=5, cutoff 0.1) based on experiments rather than tuned per task.
- Fairness and spurious-correlation experiments are described as heuristic and not the primary focus.
- Waterbirds and other results compare to a limited set of baselines and report average accuracy only partly in the main text.
Experimental setup
Datasets: Spambase Toxicity QSAR Breast Cancer Wisconsin Religious texts Waterbirds MultiNLI CivilComments CelebA NICO++ MetaShift CheXpert
Benchmarks: Spambase Toxicity QSAR Breast Cancer Wisconsin Religious texts Waterbirds MultiNLI CivilComments CelebA NICO++ MetaShift CheXpert
Metrics: test MSE logistic test loss average accuracy worst group accuracy accuracy of sensitive attribute prediction
Baselines: FOCI GenericUnivariateSelect SelectPercentile SelectFpr SelectFdr SelectFwe SelectKBest UMAP PCA LDA ERM DRO
Builds on: FOCI Chatterjee's rank correlation coefficient Group DRO Just Train Twice
Closest papers by meaning
- SAND: One-Shot Feature Selection with Additive Noise Distortion (ICML 2025 · similarity 0.84)
- Estimating Conditional Mutual Information for Dynamic Feature Selection (ICLR 2024 · similarity 0.83)
- Contextual Feature Selection with Conditional Stochastic Gates (ICML 2024 · similarity 0.83)
- Rank-Transformed Dissimilarity Profiles for High-Dimensional Classification (NeurIPS 2026 · similarity 0.83)
- Generative Classifiers Avoid Shortcut Solutions (ICLR 2025 · similarity 0.82)
- DiffuLT: Diffusion for Long-tail Recognition Without External Knowledge (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.