Research map / Large language models / Language, safety and interpretability
FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FACTER, a post hoc, black-box framework that combines conformal prediction-based fairness thresholds with violation-triggered prompt updates to reduce demographic bias in LLM-based recommendation.
Problem
LLM-based recommenders can produce demographically biased outputs, and it is unclear how to set a principled threshold for what counts as a fairness violation and how to mitigate it without retraining black-box models.
Approach
Combines conformal prediction over a fairness-aware non-conformity score (predictive embedding error plus cross-group embedding divergence) with an online loop that injects "avoid" examples of detected violations into the system prompt and decays the threshold, without updating model weights.
Key results
- On MovieLens-1M, FACTER (Iter3) reduces violations to 5 from 112 (Zero-Shot) and 28 (UP5) with NDCG@10 0.445 vs 0.427 for UP5 and 0.458 for Zero-Shot.
- On Amazon Movies & TV, violations drop to 18 (vs 198 Zero-Shot, 63 UP5) with CFR 0.634 vs 0.721 for UP5.
- Reported 95.5% violation reduction on MovieLens-1M is computed relative to the method's own initial iteration, not to the baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FACTER instead of UP5 | LLM recommender · MovieLens-1M | #Violations: 28 -> 5 | Helped |
| Apply FACTER instead of UP5 | LLM recommender · MovieLens-1M | CFR: 0.613 -> 0.591 | Helped |
| Apply FACTER instead of UP5 | LLM recommender · MovieLens-1M | NDCG@10: 0.427 -> 0.445 | Helped |
| Apply FACTER instead of UP5 | LLM recommender · Amazon Movies & TV | #Violations: 63 -> 18 | Helped |
| Apply FACTER instead of UP5 | LLM recommender · Amazon Movies & TV | CFR: 0.721 -> 0.634 | Helped |
| Apply FACTER instead of UP5 | LLM recommender · Amazon Movies & TV | NDCG@10: 0.328 -> 0.339 | Helped |
| Increase fairness penalty lambda (ablation) | FACTER · MovieLens-1M, iteration 3 | #Violations: 18 (lambda=0.1) -> 2 (lambda=0.7); 2 (lambda=0.7) at 0.9 | Helped |
| Increase fairness penalty lambda (ablation) | FACTER · MovieLens-1M, iteration 3 | NDCG@10: 0.446 (lambda=0.1) -> 0.395 (lambda=0.9) | Hurt |
| Increase fairness penalty lambda (ablation) | FACTER · Amazon, iteration 3 | #Violations: 41 (lambda=0.1) -> 12 (lambda=0.9) | Helped |
| Increase fairness penalty lambda (ablation) | FACTER · Amazon, iteration 3 | NDCG@10: 0.333 (lambda=0.1) -> 0.341 (lambda=0.5) -> 0.312 (lambda=0.9) | Mixed |
| Increase neighborhood similarity threshold tau_rho (ablation) | FACTER · MovieLens-1M, iteration 3 | #Violations: 16 -> 4 | Helped |
| Increase neighborhood similarity threshold tau_rho (ablation) | FACTER · MovieLens-1M, iteration 3 | NDCG@10: 0.442 -> 0.435 | Hurt |
Surprising findings
- The headline 95.5% reduction is relative to FACTER's own initial iteration; relative to Zero-Shot the drop is about 95.5% as well (112 to 5), but the paper's Table 1 and text are inconsistent on this framing.
- Reported theoretical Type I error bound (0.201) is far looser than the empirical rate (0.018), and the Theorem 'Threshold Update Convergence' proof is only heuristic.
Limitations
- Offline calibration is O(n^2) in pairwise similarity computations.
- Assumes the sentence embedder is relatively bias-free.
- Prompt length limits the number of avoid examples that can be injected.
Experimental setup
Datasets: MovieLens-1M Amazon Movies & TV
Benchmarks: MovieLens-1M Amazon Movies & TV
Metrics: #Violations SNSR CFR NDCG@10 Recall@10
Baselines: UP5 Zero-Shot
Builds on: conformal prediction UP5 zero-shot LLM rankers
Closest papers by meaning
- CUGF: A Reliable and Fair Recommendation Framework (AAAI 2025 · similarity 0.85)
- Bridging Jensen Gap for Max-Min Group Fairness Optimization in Recommendation (ICLR 2025 · similarity 0.84)
- Interpolating Item and User Fairness in Multi-Sided Recommendations (NeurIPS 2024 · similarity 0.84)
- Echoes in Filter Bubble: Diagnosing and Curing Popularity Bias in Generative Recommender Systems (NeurIPS 2026 · similarity 0.84)
- ENSUR: Equitable and Statistically Unbiased Recommendation (ICML 2025 · similarity 0.83)
- Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.