Research map / Large language models / Agents, code and math
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 | LLMs evaluated on LegalBench rule-application · LegalBench rule-application | correctness: 58.5 -> 82.2 | Helped |
| Use GPT-4 instead of GPT-3.5 | LLMs evaluated on LegalBench rule-application · LegalBench rule-application | analysis: 44.2 -> 79.7 | Helped |
| Use GPT-4 instead of Claude-1 | LLMs evaluated on LegalBench rule-application · LegalBench rule-application | correctness: 61.4 -> 82.2 | Helped |
| Use GPT-4 instead of Claude-1 | LLMs evaluated on LegalBench rule-application · LegalBench rule-application | analysis: 59.0 -> 79.7 | Helped |
| Use a plain-language task prompt instead of a technical-language prompt | LLMs evaluated on the studied LegalBench tasks · LegalBench prompt-engineering subset | balanced-accuracy: up to 21 points (balanced-accuracy) | Helped |
| Change the in-context demonstrations | LLMs evaluated on the studied LegalBench tasks · LegalBench prompt-engineering subset | balanced-accuracy: up to 20pts (balanced-accuracy) | Mixed |
| Provide the relevant rule in the prompt rather than relying on latent rule knowledge | LLMs on rule-conclusion tasks · LegalBench rule-conclusion prompt-engineering subset | performance | No clear effect |
| Increase Flan-T5 model size from 3B to 11B parameters | Flan-T5 · LegalBench | average performance across five reasoning categories | Helped |
| Increase LLaMA-2 model size from 7B to 13B parameters | LLaMA-2 · LegalBench | average performance across four reasoning categories | Helped |
| Use Flan-T5-XXL instead of GPT-3.5 or Claude-1 | LLMs evaluated on LegalBench · LegalBench issue-spotting | average performance | Helped |
| Use Flan-T5-XXL instead of GPT-3.5 or Claude-1 | LLMs evaluated on LegalBench · LegalBench rhetorical-understanding | average performance | Helped |
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.91)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.89)
- LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis (AAAI 2026 · similarity 0.88)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.88)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.87)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.