Research map / Learning theory and algorithms / Prediction and decision losses
RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RankSHAP, a Shapley-value feature attribution method for learning-to-rank that satisfies ranking-adapted axioms (efficiency, missingness, symmetry, monotonicity) using NDCG as the value function, and evaluates it on MS MARCO and Robust04 with a user study.
Problem
Post-hoc feature attributions for ranking models are inconsistent and often contradict each other; existing ranking attribution methods (EXS, RankLIME, RankingSHAP) do not satisfy basic Shapley-style axioms.
Approach
Define Rank-adapted Shapley axioms and a generalized ranking effectiveness metric (GREM, a weighted sum of gains times position discounts) as the value function, so that the Shapley value (approximated via KernelSHAP with NDCG) gives a unique, axiomatically grounded ranking attribution.
Key results
- RankSHAP surpasses the best competitor by 25.78% on Fidelity and 19.68% on wFidelity averaged across rankers, datasets, and document settings.
- On MS MARCO with BM25 top-10, RankSHAP reaches Fidelity 0.63 vs 0.52 for RankingSHAP.
- In a 30-participant user study, RankSHAP outperformed the strongest baseline by 19.1% on passage reordering (Kendall's Tau 0.56 vs 0.52 for RankingSHAP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RankSHAP instead of the strongest competing attribution method | BM25, BERT, T5, and LLAMA2 rankers · MS MARCO and Robust04, Top-10 documents | Fidelity: RankSHAP surpasses the top-performing competitor by $25.78\%$ on Fidelity | Helped |
| Use RankSHAP instead of the strongest competing attribution method | BM25, BERT, T5, and LLAMA2 rankers · MS MARCO and Robust04, Top-10 documents | wFidelity: $19.68\%$ | Helped |
| Use RankSHAP attributions rather than the strongest baseline attributions | BERT-ranked passages in the user study · User study, passage reordering task (Q1) | Fidelity (Kendall's Tau): 0.52 -> 0.56 | Helped |
| Use RankSHAP attributions rather than the strongest baseline attributions | BERT-ranked passages in the user study · User study, query estimation task (Q2) | GPT-4 similarity: 0.58 -> 0.69 | Helped |
| Generate attributions for more documents (10 to 20) | RankSHAP · MS MARCO and Robust04 | Fidelity: 20\% average performance drop | Hurt |
| Generate attributions for more documents (20 to 100) | RankSHAP · MS MARCO and Robust04 | Fidelity: 14.6\% average decrease | Hurt |
| Use neural BERT or T5 rather than BM25 as the ranker | RankSHAP · MS MARCO and Robust04 | Fidelity: 13.3\% and $14.7\%$ lower | Hurt |
Surprising findings
- The paper reports that random attributions obtain a concordance of 0.23 in the user study, higher than their metric-evaluated Fidelity, suggesting participants rely on prior topical beliefs.
Limitations
- Assumes feature independence inherent to Shapley values.
- Requires relevance labels or proxies (e.g., BM25 or implicit signals) for the value function.
- KernelSHAP is perturbation-based and results may depend on sampling density.
Experimental setup
Datasets: MS MARCO Robust04
Benchmarks: MS MARCO Robust04
Metrics: Fidelity wFidelity
Baselines: Random EXS RankLIME RankingSHAP
Builds on: KernelSHAP LIME RankingSHAP RankLIME EXS
Closest papers by meaning
- WeightedSHAP: analyzing and improving Shapley based feature attributions (NeurIPS 2022 · similarity 0.91)
- L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data (ICLR 2019 · similarity 0.88)
- shapiq: Shapley Interactions for Machine Learning (NeurIPS 2024 · similarity 0.87)
- OrdShap: Feature Position Importance for Sequential Black-Box Models (NeurIPS 2025 · similarity 0.85)
- Foundations of the Theory of Performance-Based Ranking (CVPR 2025 · similarity 0.85)
- Unlocking the Game: Estimating Games in Möbius Representation for Explanation and High-Order Interaction Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.