Research map / Learning theory and algorithms / Fairness
FairJob: A Real-World Dataset for Fairness in Online Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases FairJob, a real-world, anonymized job-advertising click dataset with a proxy for gender, and benchmarks fairness-utility trade-offs of simple bias mitigation methods on it.
Problem
Fairness research in online advertising lacks public, realistic tabular datasets, and protected attributes such as gender are usually unavailable due to privacy constraints, making fairness evaluation of job ad recommendation hard.
Approach
Provides a large anonymized job-campaign click dataset in which gender is replaced by a behavioral proxy (dominant gender of products a user interacts with), plus a selection-bias-corrected click-rank utility metric for evaluating recommendation fairness on biased logs.
Key results
- On FairJob, the unaware XGBoost reaches AUC 0.758 and the unfair XGBoost AUC 0.762, while the Dummy classifier has AUC 0.500 with DP 0.00000.
- Adding a fairness penalty to logistic regression (fairness_multiplier 0.5 and 1.0) reduces DP while giving slight utility gains, especially on the selection-bias-corrected utility U tilde, relative to the unfair model.
- The dataset has 1,072,226 rows over 5 months of a job campaign, with a positive click rate of about 0.7% (7,489 clicks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include the protected attribute as a feature | XGBoost · FairJob | NLLH: 0.05491 -> 0.05736 | Hurt |
| include the protected attribute as a feature | XGBoost · FairJob | AUC: 0.75787 -> 0.76201 | Helped |
| include the protected attribute as a feature | XGBoost · FairJob | DP: 0.00278 -> 0.00323 | Hurt |
| include the protected attribute as a feature | XGBoost · FairJob | U: 0.01017 -> 0.01037 | Helped |
| include the protected attribute as a feature | XGBoost · FairJob | tilde U: 0.01276 -> 0.01236 | Hurt |
| increase the fairness multiplier above zero (ablation) | Logistic Regression fair model · FairJob | DP | Helped |
| increase the fairness multiplier above zero (ablation) | Logistic Regression fair model · FairJob | NLLH | Hurt |
| set fairness_multiplier to 0.5 or 1.0 (ablation) | Logistic Regression fair model · FairJob | tilde U | Helped |
| apply the fairness penalty to only a percentage of the training set (ablation) | Fairness-penalized model · FairJob | fairness | Helped |
Surprising findings
- The Dummy classifier is perfectly fair on demographic parity yet its utility metrics are close to XGBoost, which the authors attribute to strong class imbalance.
- The unfair model does not clearly dominate the unaware model; the unaware XGBoost gets the best U tilde and NLLH despite not seeing the protected attribute.
Limitations
- The gender proxy is a behavioral signal (dominant gender of products interacted with), not true gender, and its accuracy cannot be verified.
- Data is sub-sampled non-uniformly and features are anonymized and randomly projected, limiting interpretability and reuse.
- The senior-position label is derived heuristically from product titles and may be noisy.
Experimental setup
Datasets: FairJob
Benchmarks: FairJob
Metrics: NLLH AUC-ROC demographic parity (DP) click-rank utility U product-rank utility for biased data (U tilde)
Baselines: Dummy classifier XGBoost (unaware) XGBoost (unfair) Logistic regression (unfair) Logistic regression (unaware)
Builds on: fairness through unawareness demographic parity click-rank utility
Closest papers by meaning
- FairLISA: Fair User Modeling with Limited Sensitive Attributes Information (NeurIPS 2023 · similarity 0.87)
- Interpolating Item and User Fairness in Multi-Sided Recommendations (NeurIPS 2024 · similarity 0.87)
- Reconciling Predictive and Statistical Parity: A Causal Approach (AAAI 2024 · similarity 0.86)
- Fairness-Accuracy Trade-Offs: A Causal Perspective (AAAI 2025 · similarity 0.85)
- Fairness under Competition (NeurIPS 2025 · similarity 0.85)
- User-item fairness tradeoffs in recommendations (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.