Research map / Probabilistic and scientific modeling / Causal discovery
Counterfactual Voting Adjustment for Quality Assessment and Fairer Voting in Online Platforms with Helpfulness Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Counterfactual Voting Adjustment (CVA), a causal backdoor-adjusted voting model that jointly corrects position and herding biases in helpfulness votes to estimate fairer response quality on StackExchange.
Problem
Helpfulness votes on user-generated responses are biased by display position and by herding on prior votes, so aggregated votes misrepresent true content quality.
Approach
Frame per-vote behavior as a causal problem with the response's presentational features as the confounder, and estimate counterfactual votes by integrating out existing vote distribution and display rank (backdoor adjustment) within a logistic voting model with per-answer quality parameters.
Key results
- On the politics community with comment sentiment as proxy, CVA achieves residual 7776 versus 8102 for CVP and 9691 for voteDiff.
- With GPT-4o helpfulness as ground truth on politics, CVA residual is 6865 versus 7463 (CVP) and 9115 (voteDiff).
- CVA residuals beat both voteDiff and CVP in 94 of the 120 communities (78.3%) under the comment-sentiment proxy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CVA instead of CVP | response-quality ranking · Politics (real, comment sentiment) | sum of squared residuals: 8102 -> 7776 | Helped |
| Use CVA instead of voteDiff | response-quality ranking · Politics (real, comment sentiment) | sum of squared residuals: 9691 -> 7776 | Helped |
| Use CVA instead of CVP | response-quality ranking · Politics (real, GPT-4o helpfulness) | sum of squared residuals: 7463 -> 6865 | Helped |
| Use CVA instead of voteDiff | response-quality ranking · Politics (real, GPT-4o helpfulness) | sum of squared residuals: 9115 -> 6865 | Helped |
| Use CVA instead of voteDiff and CVP | response-quality ranking across 120 StackExchange communities · Real data, comment sentiment | communities with smaller residual than both baselines: 94 (78.3\%) | Helped |
| Use CVA instead of voteDiff and CVP | response-quality ranking across 120 StackExchange communities · Real data, comment sentiment | communities with higher Kendall’s tau: 89 (74.2\%) | Helped |
| Use CVA instead of voteDiff and CVP | response-quality ranking across 120 StackExchange communities · Real data, GPT-4o helpfulness | communities with smaller residual than both baselines: 93 (77.5\%) | Helped |
| Use CVA instead of voteDiff and CVP | response-quality ranking across 120 StackExchange communities · Real data, GPT-4o helpfulness | communities with higher Kendall’s tau than both baselines: 90 (75\%) | Helped |
| Jointly adjust for display position and prior-vote distribution | CVA · Semi-synthetic data, 8 representative communities | Kendall’s tau rank correlation with true quality | Helped |
| Jointly adjust for display position and prior-vote distribution | CVA · Real data, 8 representative communities (comment sentiment) | Kendall’s tau rank correlation with true quality | Helped |
| Jointly adjust for display position and prior-vote distribution | CVA · Real data, GPT-4o helpfulness | ranking alignment with GPT-4o evaluation | Mixed |
Surprising findings
- CVA does not always significantly outperform CVP; advantage is small in communities with little position bias (e.g., some meta communities).
Limitations
- Relies on backdoor adjustment assumptions; only relative response length is used as pre-reading feature, so unobserved confounders may remain.
- Uses GPT-4o helpfulness and comment sentiment as imperfect proxies for true quality since ground truth is unknown.
- Perceived quality assumed to affect only the next vote; reading-without-voting events are excluded.
Experimental setup
Datasets: StackExchange data dump (Oct 5, 2022)
Benchmarks: semi-synthetic StackExchange data (8 communities) real StackExchange data with comment sentiment proxy real StackExchange data with GPT-4o helpfulness proxy
Metrics: Kendall's tau sum of squared residuals to diagonal win rate across communities
Baselines: vote difference ranking (voteDiff) Chinese Voting Process (CVP)
Builds on: Chinese Voting Process backdoor adjustment
Closest papers by meaning
- Representative Ranking for Deliberation in the Public Sphere (ICML 2025 · similarity 0.82)
- Computing Voting Rules with Improvement Feedback (ICML 2025 · similarity 0.82)
- Alleviating Dual Biases in Recommendation (Student Abstract) (AAAI 2025 · similarity 0.82)
- Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses (NeurIPS 2026 · similarity 0.81)
- Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings (NeurIPS 2025 · similarity 0.81)
- Selective Preference Aggregation (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.