Research map / Large language models / Language, safety and interpretability
When Neutral Summaries Are Not That Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures political neutrality of LLM default news summaries on five US hot-button issues by comparing them to pro-Democrat and pro-Republican summaries using classifier-separability and vocabulary analysis, finding a consistent pro-Democratic tilt.
Problem
Whether LLMs produce politically neutral summaries of polarizing news is unquantified; prior work mostly uses qualitative or binary bias assessments and lacks comparative quantitative analysis across models.
Approach
Quantify neutrality via a Polarization Index: the difference in classifier accuracy (SentenceTransformer-based) separating default summaries from Democrat-aligned versus Republican-aligned summaries, combined with a diverging-vocabulary token bias score and a cross-model consistency index for algorithmic monoculture.
Key results
- Gun control/rights shows the largest max polarization index (-9.49% for PaLM-2), with healthcare next (-6.14% for LLaMA-7B).
- Average consistency index for Democrat-leaning vocabulary is 55.37% and for Republican-leaning 52.49% across models.
- Cross-model classifier accuracy averages 52.98% (Democrat) and 53.47% (Republican) versus 55.45% and 57.99% for same-model transfer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PaLM-2 rather than the other evaluated models for default news summaries | LLM-generated news summaries · Gun Control/Rights | Polarization Index: -9.49% | Mixed |
| Use LLaMA-7B rather than the other evaluated models for default news summaries | LLM-generated news summaries · Across five political topics | Average Polarization Index: -4.86% | Mixed |
| Generate default summaries across the evaluated models and topics | LLM-generated news summaries · Five sociopolitical topics | Polarization Index: -2.83% | Mixed |
| Compare Democrat-leaning vocabulary across models | The four evaluated LLMs · Political news summaries | Vocabulary overlap: 55.37% | Mixed |
| Apply classifiers across different LLMs instead of evaluating on the same model | SentenceTransformer-based classifiers · Default versus Democrat-aligned summaries | Classification accuracy: 55.45% to 52.98% | Mixed |
| Apply classifiers across different LLMs instead of evaluating on the same model | SentenceTransformer-based classifiers · Default versus Republican-aligned summaries | Classification accuracy: 57.99% to 53.47% | Mixed |
Surprising findings
- Vicuna-7B shows a slight pro-Republican tilt (+1.33%) on gun control/rights, contrary to the overall pro-Democratic trend.
Limitations
- Excludes larger models such as GPT-4 and GPT-3.5 due to computational cost.
- Only four models evaluated, all relatively small except PaLM-2; appendix and prompts deferred to supplementary material.
- Classifier-accuracy-based polarization index is an indirect proxy and differences are small (near chance levels for many cases).
Experimental setup
Datasets: GNews-scraped US news articles on five hot-button issues (2019-2024)
Metrics: Polarization Index (classifier accuracy difference) token bias score B(t) consistency index CI cross-model classifier accuracy A
Baselines: PaLM-2 Vicuna-7B Mistral-7B LLaMA-7B
Builds on: algorithmic monoculture (Kleinberg & Raghavan) classification accuracy as proxy for text separability
Closest papers by meaning
- Quantifying Political Polarization through the Lens of Machine Translation and Vicarious Offense (AAAI 2024 · similarity 0.86)
- Examining Alignment of Large Language Models through Representative Heuristics: the case of political stereotypes (ICLR 2025 · similarity 0.86)
- Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses (NeurIPS 2026 · similarity 0.86)
- Innocuous-Seeming Data, Latent Ideology: Ideological Generalisation in Finetuned LLMs (NeurIPS 2026 · similarity 0.85)
- Linear Representations of Political Perspective Emerge in Large Language Models (ICLR 2025 · similarity 0.85)
- Political Bias Prediction Models Focus on Source Cues, Not Semantics (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.