Research map / Multi-agent systems and games / Games and equilibria
Carrot and Stick: Eliciting Comparison Data and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Eliciting truthful pairwise comparison data (e.g., for RLHF or ranking) from agents without ground-truth verification, where comparison tasks are not ex-ante identical and so multi-task peer prediction does not apply; also eliciting networked binary data with homophily.
Approach
Identifies "uniform dominance" as an information-structure condition under which the bonus-penalty payment is the unique payment inducing strict truthful BNE; shows Bayesian SST comparison models satisfy it via assigning three pairs (a,a'), (a'',a'), (a'',a), and extends to Ising-model networks by pairing friends and non-friends.
Key results
- On the SUSHI dataset, over 75% of users receive positive payments under truth-telling versus about 50% under uninformed random reporting.
- On Last.fm (Lady Gaga), more than 76% of agents get positive payments with average payment 0.37 under truth-telling, while random reporting yields near-zero average payment.
- Under unilateral deviation to random reports on Last.fm, the payment drops from 0.37 to near zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| have all agents report comparisons truthfully rather than use uniform random reports | BPP mechanism · SUSHI preference dataset | fraction receiving positive payments: about 50% -> over 75% | Helped |
| have all agents report signals truthfully rather than use iid random reports | BPP mechanism · Last.fm, Lady Gaga | average payment: near zero -> 0.37 | Helped |
| have one agent report listener/non-listener uniformly at random while others are truthful | BPP mechanism · Last.fm, Lady Gaga | average payment: $0.37$ to near zero | Hurt |
| have one agent replace her comparisons with uniform random bits while others are truthful | BPP mechanism · SUSHI preference dataset | payment | Hurt |
Limitations
- Assumes agents are a priori similar; may fail under polarized preferences.
- Truth-telling is only symmetric-strongly-truthful; non-symmetric equilibria not ruled out.
- Experiments use datasets that only approximately satisfy the theoretical assumptions (e.g., SUSHI restricted subset, biased Last.fm listener fractions).
Experimental setup
Datasets: SUSHI preference dataset Last.fm dataset
Benchmarks: SUSHI preference dataset Last.fm dataset
Metrics: empirical cumulative distribution of payments fraction of agents with positive payment average payment
Baselines: agreement mechanism multi-task peer prediction bonus-penalty mechanism
Builds on: multi-task peer prediction bonus-penalty payment agreement mechanism Ising model
Closest papers by meaning
- Stochastically Dominant Peer Prediction (NeurIPS 2025 · similarity 0.89)
- A Cramér–von Mises Approach to Incentivizing Truthful Data Sharing (NeurIPS 2025 · similarity 0.88)
- Incentivizing Truthful Language Models via Peer Elicitation Games (NeurIPS 2025 · similarity 0.88)
- Plant-and-Steal: Truthful Fair Allocations via Predictions (NeurIPS 2024 · similarity 0.85)
- You Get What You Give: Reciprocally Fair Federated Learning (ICML 2025 · similarity 0.85)
- Private Blotto: Viewpoint Competition with Polarized Agents (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.