Research map / Large language models / Instruction and fine-tuning
OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenDebateEvidence, a 3.5M-document dataset of American competitive debate evidence, and benchmarks LoRA, ReFT, and orthogonalization fine-tuning of LLaMA3 and Mistral models for argumentative summarization.
Problem
Existing argument mining and summarization datasets from competitive debate are small or limited to pre-season evidence, lacking large-scale regular-season debate evidence with rich metadata.
Approach
Mines the OpenCaseList corpus of open-sourced debate evidence (.docx cards) into a large, metadata-rich dataset with hierarchical tags/hats/pockets and benchmarks parameter-efficient fine-tuning for abstractive card-tag summarization.
Key results
- LoRA fine-tuned LLaMA3-70B reaches ROUGE-1 37.2% on OpenDebateEvidence versus 33.8% for base LLaMA3-70B and 32.5% for base Gemini.
- LoRA fine-tuned LLaMA3-8B with 1M examples achieves 87.1% accuracy on stance detection, the highest among reported models.
- Pretraining on OpenDebateEvidence improves LLaMA3-8B ArgAnalysis35K ROUGE-1 from 42.4% to 48.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA3-70B with LoRA rather than use the base model (ablation) | LLaMA3-70B · OpenDebateEvidence | ROUGE-1 F1 (%): 33.8 ± 1.2 -> 37.2 ± 1.0 | Helped |
| Use LoRA rather than ReFT (ablation) | LLaMA3-70B · OpenDebateEvidence | ROUGE-1 F1 (%): 35.9 ± 1.1 -> 37.2 ± 1.0 | Helped |
| Use LoRA rather than Orthogonal (ablation) | LLaMA3-70B · OpenDebateEvidence | ROUGE-1 F1 (%): 34.2 ± 1.2 -> 37.2 ± 1.0 | Helped |
| Fine-tune LLaMA3-70B with LoRA rather than use the base model (ablation) | LLaMA3-70B · BillSum | ROUGE-1 F1 (%): 50.2 ± 1.1 -> 54.6 ± 1.0 | Helped |
| Use LoRA rather than ReFT (ablation) | LLaMA3-70B · BillSum | ROUGE-1 F1 (%): 52.9 ± 1.1 -> 54.6 ± 1.0 | Helped |
| Use LoRA rather than Orthogonal (ablation) | LLaMA3-70B · BillSum | ROUGE-1 F1 (%): 51.1 ± 1.2 -> 54.6 ± 1.0 | Helped |
| Fine-tune LLaMA3-70B with LoRA rather than use the base model (ablation) | LLaMA3-70B · DebateSum | ROUGE-1 F1 (%): 36.7 ± 1.2 -> 39.4 ± 1.1 | Helped |
| Use LoRA rather than ReFT (ablation) | LLaMA3-70B · DebateSum | ROUGE-1 F1 (%): 38.1 ± 1.2 -> 39.4 ± 1.1 | Helped |
| Use LoRA rather than Orthogonal (ablation) | LLaMA3-70B · DebateSum | ROUGE-1 F1 (%): 36.9 ± 1.3 -> 39.4 ± 1.1 | Helped |
| Fine-tune LLaMA3-70B with LoRA rather than use the base model (ablation) | LLaMA3-70B · xSum Validation | ROUGE-1 F1 (%): 46.5 ± 0.7 -> 48.2 ± 0.6 | Helped |
| Use LoRA rather than ReFT (ablation) | LLaMA3-70B · xSum Validation | ROUGE-1 F1 (%): 47.9 ± 0.7 -> 48.2 ± 0.6 | Helped |
| Train LLaMA3-8B with LoRA on 1M examples instead of the standard LoRA setting (ablation) | LLaMA3-8B · OpenDebateEvidence | ROUGE-1 F1 (%): 25.7 ± 0.7 -> 32.2 ± 1.5 | Helped |
Surprising findings
- Orthogonalization gave only modest gains compared with LoRA and ReFT across tasks.
- Neural semantic deduplication removed 77.4% of documents, although the authors argue many 'duplicates' still carry distinct metadata value.
Limitations
- Dataset is dominated by American high school and college debate and English-language evidence, limiting representativeness.
- Public Forum evidence is a small fraction of the dataset.
- Metadata and tags may be inconsistent, and some examples contain academic dishonesty or noise.
Experimental setup
Datasets: OpenDebateEvidence DebateSum BillSum xSum ArgAnalysis35K Multi-LexSum
Benchmarks: OpenDebateEvidence DebateSum BillSum xSum ArgAnalysis35K Multi-LexSum
Metrics: ROUGE-1 ROUGE-2 ROUGE-L perplexity LLM-as-judge quality score accuracy F1
Baselines: Mistral-7B base LLaMA3-8B base LLaMA3-70B base Google Gemini base Anthropic Claude base
Builds on: DebateSum LoRA ReFT PEFT
Closest papers by meaning
- Corpus Wide Argument Mining—A Working Solution (AAAI 2020 · similarity 0.90)
- Strategic Planning and Rationalizing on Trees Make LLMs Better Debaters (ICLR 2026 · similarity 0.84)
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.84)
- Debating with More Persuasive LLMs Leads to More Truthful Answers (ICML 2024 · similarity 0.83)
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.83)
- An Extension-Based Argument-Ranking Semantics: Social Rankings in Abstract Argumentation (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.