Research map / Learning theory and algorithms / Fairness

Assessing Uncertainty in Similarity Scoring: Performance & Fairness in Face Recognition

ICLR 2024 · Learning theory and algorithms › Fairness · approach: Post-processing and transport for fairness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that naive bootstrap underestimates similarity ROC curves in face recognition because FAR/FRR are U-statistics, and proposes a recentered bootstrap to build valid confidence bands for ROC and fairness metrics.

Problem

Uncertainty quantification for empirical similarity ROC curves and FR fairness metrics is poorly handled; FAR/FRR in similarity scoring are U-statistics, so the naive bootstrap gives biased, invalid confidence bands.

Approach

A recentered bootstrap for pairwise similarity ROC curves: recenter bootstrap replicates around the V-statistic version of the ROC (which removes the diagonal-term bias), yielding asymptotically valid confidence bands, plus a normalized scalar uncertainty measure for ROC and fairness metrics.

Key results

• On synthetic vMF embeddings, the recentered bootstrap achieves near-nominal coverage at FAR=1e-5 (e.g., 0.96 at nominal 0.95) and at FAR=1e-1 (0.92 at nominal 0.95), while the naive bootstrap reaches only 0.55 at nominal 0.95.
• The max-geomean fairness metric shows the lowest normalized uncertainty among the compared FR fairness metrics, in both FAR and FRR, on MORPH and RFW experiments.

Measured effects reported in the paper

Surprising findings

• The naive bootstrap confidence bands for the ROC often do not even contain the empirical ROC curve, and have estimated coverage near zero at FAR=1e-5.
• Two ArcFace/CosFace models that look different on one MORPH split are statistically indistinguishable once uncertainty is considered, so empirical ROC rankings can flip across datasets.

Limitations

• Naive bootstrap is invalid for pairwise similarity statistics; the recentering is needed and theory relies on asymptotic assumptions.
• Coverage validation is mainly on synthetic vMF embeddings rather than real face data.
• Experiments use pretrained open-source FR models and only gender, age, and race as sensitive attributes.

Experimental setup

Datasets: MS-Celeb-1M-v1c-r MORPH RFW MS1M-RetinaFace

Benchmarks: MORPH RFW

Metrics: empirical ROC confidence band coverage normalized uncertainty max-min ratio max-geomean ratio log-geomean sum Gini coefficient

Baselines: naive bootstrap gaussian approximation of recentered bootstrap

Builds on: naive bootstrap (Bickel-Freedman style) NIST FRVT fairness metrics generalized U-statistics

Closest papers by meaning

• Standardized Interpretable Fairness Measures for Continuous Risk Scores (ICML 2024 · similarity 0.83)
• A Sequentially Fair Mechanism for Multiple Sensitive Attributes (AAAI 2024 · similarity 0.82)
• FROC: Building Fair ROC from a Trained Classifier (AAAI 2025 · similarity 0.82)
• Kernel-based Equalized Odds: A Quantification of Accuracy-Fairness Trade-off in Fair Representation Learning (NeurIPS 2025 · similarity 0.82)
• The VOROS: Lifting ROC Curves to 3D to Summarize Unbalanced Classifier Performance (AAAI 2025 · similarity 0.82)
• FAIR-FER: A Latent Alignment Approach for Mitigating Bias in Facial Expression Recognition (Student Abstract) (AAAI 2024 · similarity 0.81)