Research map / Large language models / AI and society
Measuring And Improving Persuasiveness Of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces transsuasion (rewriting low-engagement tweets into high-engagement ones) built from natural experiments on 1.57M Twitter pairs, plus PersuasionBench/PersuasionArena to measure LLM persuasiveness, and shows a fine-tuned 13B Vicuna can beat GPT-4 on these tasks.
Problem
Measuring and improving the persuasiveness of LLM-generated content automatically, isolated from sender, receiver, time and channel, since human persuasion studies are costly and confounded.
Approach
Uses natural experiments (semantically similar tweets from the same brand account posted close in time with different like counts) to build transsuasion pairs, and fine-tunes a small model on behavior simulation, content simulation, transsuasion tasks and synthetic explanations so it outperforms much larger models on persuasion benchmarks.
Key results
- Ours (CS+BS+TS) 13B reaches 80.9% accuracy on comparative transsuasion random split vs 54.5%... GPT-4o 62.1% (10-shot) and GPT-4 56.2% (10-shot).
- In oracle-judged generative transsuasion, the 13B instruct model achieves an average Elo of 1299-1304 vs GPT-4 at 1213 and GPT-4o at 1251.
- On marketing blog simulation, the 13B Ours-Instruct gets 68.8 views accuracy and 50.9 dwell time accuracy, close to GPT-4 10-shot (70.4, 50.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add transsuasion training to content and behavior simulation training (ablation) | Vicuna-1.5-13B · Comparative Transsuasion Random | accuracy: 77.9 -> 80.9 | Helped |
| Add transsuasion training to content and behavior simulation training (ablation) | Vicuna-1.5-13B · Comparative Transsuasion Brand | accuracy: 76.1 -> 77.3 | Helped |
| Add transsuasion training to content and behavior simulation training (ablation) | Vicuna-1.5-13B · TS-GT Parap | Elo: 1205 -> 1271 | Helped |
| Add synthetic explanations of why the higher-engagement tweet performs better (ablation) | Vicuna-1.5-13B trained on CS+BS+TS · TS-GT average | Elo: 1293 -> 1299 | Helped |
| Iterate transsuasion generation 3 times (ablation) | Ours (13B) · TS-GT average | Elo: 1293 -> 1304 | Helped |
| Increase training from 0.50 ep to 1.00 ep (ablation) | Ours (CS+BS+TS) (13B) · Comparative Transsuasion Random | accuracy: 73.3 -> 80.9 | Helped |
| Increase the model size from 7B to 13B (ablation) | Ours (CS+BS+TS) · Comparative Transsuasion Random | accuracy: 74.1 -> 80.9 | Helped |
| Train on content simulation and behavior simulation only, excluding transsuasion (ablation) | Vicuna-1.5-13B · TS-GT Parap | Elo: 1271 -> 1205 | Hurt |
Surprising findings
- Expert marketers' allocated ad budget shows no meaningful correlation with ad KPIs, suggesting humans poorly predict persuasion.
- Parameter-count / FLOP-based regulation metrics fail to capture persuasive capability, since a much smaller fine-tuned model outperforms GPT-4.
Limitations
- Oracle judge is itself an LLM fine-tuned on the same data, so evaluation may be biased toward the training distribution.
- Data is limited to enterprise Twitter accounts and a few human studies, limiting generality.
- Ethical risks of misuse for misinformation and manipulation.
Experimental setup
Datasets: Twitter brand tweets (1.57M transsuasion pairs, from 180M collected tweets) Fortune 500 marketing blog posts (2,187) Fortune 500 ad budget/KPI data (>1000 ads) LLM-generated social caption feedback from 20,000+ users
Benchmarks: PersuasionBench PersuasionArena
Metrics: accuracy BLEU ROUGE BERTScore Elo rating Spearman rank correlation cumulative probability
Baselines: GPT-3.5 GPT-4 GPT-4o LLaMA-3-70B Vicuna-1.5-13B
Builds on: Vicuna-1.5 LMSYS Chatbot Arena Elo rating
Closest papers by meaning
- Debating with More Persuasive LLMs Leads to More Truthful Answers (ICML 2024 · similarity 0.86)
- Are Large Language Models Sensitive to the Motives Behind Communication? (NeurIPS 2025 · similarity 0.86)
- Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.86)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.86)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.85)
- AI-Mediated Communication Can Steer Collective Opinion (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.