Research map / Large language models / Language, safety and interpretability
The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates question-specific synthetic comments with Mistral-7B to fine-tune BERT for German political stance detection, then uses synthetic data as a reference to select the most informative unlabelled samples for human labelling.
Problem
Stance detection in online political discussions needs large labelled datasets, but the wide variety of debate topics makes labelled data scarce; LLMs used directly for stance classification are inconsistent, biased and vulnerable to attacks.
Approach
Use an offline LLM (Mistral-7B) to generate balanced favor/against synthetic comments per question to fine-tune a lightweight BERT stance classifier, and reuse these synthetic samples as a reference for a synthetic query-by-committee (SQBC) strategy that picks ambiguous unlabelled real samples for annotation.
Key results
- Baseline+Synth with M=1000 synthetic comments raises average F1 from 0.693 to 0.723 across 10 X-Stance questions.
- SQBC+Synth (M=1000) reaches average F1 0.754, surpassing the True Labels baseline (0.727) while using only 25% of labelled data.
- Zero-shot Mistral-7B reaches only 0.419 F1 and fine-tuned Mistral-7B drops to 0.182 on X-Stance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add question-aligned synthetic data to the baseline | BERT stance detection · X-Stance (avg. 10 questions) | F1: 0.693 -> 0.723 | Helped |
| Increase question-aligned synthetic data from M=200 to M=1000 (ablation) | Baseline+Synth · X-Stance (avg. 10 questions) | F1: 0.712 -> 0.723 | Helped |
| Add synthetic data to the true-label model | True Labels · X-Stance (avg. 10 questions) | F1: 0.727 -> 0.770 | Helped |
| Use synthetic data misaligned with the posed question instead of aligned synthetic data (ablation) | Baseline+Synth · X-Stance (avg. 10 questions) | F1: 0.723 -> 0.694 | Hurt |
| Replace BERT with zero-shot Mistral-7B stance classification (ablation) | Stance detection · X-Stance (avg. 10 questions) | F1: 0.693 -> 0.419 | Hurt |
| Fine-tune Mistral-7B directly for stance classification (ablation) | Stance detection · X-Stance (avg. 10 questions) | F1: 0.693 -> 0.182 | Hurt |
| Use SQBC rather than random selection with synthetic data (ablation) | Synthetic-data active learning, M=200 and 25% labels · X-Stance (avg. 10 questions) | F1: 0.720 -> 0.723 | Helped |
| Use SQBC rather than random selection with synthetic data (ablation) | Synthetic-data active learning, M=500 and 25% labels · X-Stance (avg. 10 questions) | F1: 0.735 -> 0.726 | Hurt |
| Use SQBC rather than random selection with synthetic data (ablation) | Synthetic-data active learning, M=1000 and 25% labels · X-Stance (avg. 10 questions) | F1: 0.746 -> 0.737 | Hurt |
| Combine synthetic data with manually labeled selected samples rather than train only on selected samples (ablation) | SQBC · X-Stance (avg. 10 questions), M=1000 and 25% labels | F1: 0.698 -> 0.737 | Helped |
| Increase the manually labeled share from 10% to 75% with synthetic data (ablation) | SQBC+Synth, M=1000 · X-Stance (avg. 10 questions) | F1: 0.721 -> 0.747 | Helped |
| Use SQBC+Synth with 25% labeled data instead of the all-true-label baseline | BERT stance detection · X-Stance (avg. 10 questions) | F1: 0.693 -> 0.737 | Helped |
Surprising findings
- Fine-tuning Mistral-7B on X-Stance training data worsened performance further (F1 0.182), below its zero-shot result.
- Random selection outperformed both SQBC and CAL active learning at M=1000 synthetic samples in the Synth setting.
Limitations
- A separate model is fine-tuned per question, so scaling to many topics is not addressed.
- Evaluated only on 10 selected questions of a single German dataset (X-Stance).
- Synthetic comments are more verbose and less emotional than real comments, indicating distribution differences and possible bias.
Experimental setup
Datasets: X-Stance (German) synthetic comments generated by Mistral-7B
Benchmarks: X-Stance (German)
Metrics: F1 score
Baselines: Baseline BERT True Labels Random+Synth CAL+Synth zero-shot Mistral-7B fine-tuned Mistral-7B
Builds on: query by committee contrastive active learning X-Stance
Closest papers by meaning
- Linear Representations of Political Perspective Emerge in Large Language Models (ICLR 2025 · similarity 0.84)
- DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents (NeurIPS 2026 · similarity 0.84)
- Examining Alignment of Large Language Models through Representative Heuristics: the case of political stereotypes (ICLR 2025 · similarity 0.84)
- Innocuous-Seeming Data, Latent Ideology: Ideological Generalisation in Finetuned LLMs (NeurIPS 2026 · similarity 0.83)
- When Neutral Summaries Are Not That Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries (Student Abstract) (AAAI 2025 · similarity 0.83)
- Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.