Research map / Large language models / Language, safety and interpretability
Lawma: The Power of Specialization for Legal Annotation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CaselawQA, a 260-task legal annotation benchmark built from the Supreme Court and Courts of Appeals databases, and shows lightly fine-tuned open models (Lawma 135M to 70B) outperform commercial LLMs.
Problem
Legal annotation and classification is a costly bottleneck in empirical legal research, and it is unclear how well prompted large language models perform on such tasks compared with specialized models.
Approach
Construct 260 multiple-choice legal classification tasks from the SCDB and USCAD labels, then fine-tune small open-weights models jointly on all tasks; even the 135M model beats commercial zero-shot LLMs, and a few hundred labeled examples per task suffice to match them.
Key results
- Lawma 135M reaches 83% accuracy on CaselawQA, above Claude 3.7 Sonnet at 78%.
- Lawma 70B achieves 88% accuracy, at least 10 points above every commercial model evaluated.
- With 1000 or fewer labeled examples per task, fine-tuned Llama 3.1 8B matches or exceeds Claude 3.7 Sonnet on 7 of 10 highlighted tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune a small open model on the CaselawQA annotation tasks instead of using a general commercial model | Legal annotation models · CaselawQA | accuracy: 78% to 83% | Helped |
| Increase the fine-tuned model size from 135M to 70B | Lawma · CaselawQA | accuracy: 83% to 88% | Helped |
| Fine-tune on Court of Appeals tasks | Llama 3.1 8B Instruct · Supreme Court tasks | accuracy: 50% to 64% | Helped |
| Fine-tune on the precise target annotation tasks rather than only Court of Appeals tasks | Llama 3.1 8B Instruct · Supreme Court tasks | accuracy: 25 percentage points | Helped |
| Use chain-of-thought prompting instead of few-shot direct QA (ablation) | Llama 3 Instruct models with at least 8B parameters · CaselawQA | accuracy | Helped |
| Add in-context examples with few-shot direct QA (ablation) | Llama 3 Instruct models with at most 3B parameters · CaselawQA | accuracy | Helped |
| Add in-context examples (ablation) | Llama 3.3 70B Instruct · CaselawQA | accuracy | No clear effect |
| Fine-tune Mixtral 8x7B Instruct on legal text to produce SaulLM 54B (ablation) | Mixtral 8x7B Instruct · CaselawQA | accuracy: +4 accuracy points | Helped |
| Train on increasing numbers of labeled examples (ablation) | Llama 3.1 8B Instruct fine-tuned per task · 10 highlighted CaselawQA tasks | accuracy | Helped |
| Fine-tune on all 260 tasks for one epoch instead of specializing from scratch on one task (ablation) | Llama 3 8B Instruct · 10 highlighted CaselawQA tasks | accuracy | Helped |
| Further fine-tune the broadly trained model on one task (ablation) | Llama 3 8B Instruct fine-tuned on all tasks for one epoch · 10 highlighted CaselawQA tasks | accuracy | Helped |
| Fine-tune on all tasks for three epochs instead of one epoch or one epoch plus single-task specialization (ablation) | Llama 3.1 8B Instruct · 10 highlighted CaselawQA tasks | accuracy | Helped |
Surprising findings
- The smallest fine-tuned model (135M) outperforms commercial models that are orders of magnitude larger.
- Fine-tuning on Court of Appeals tasks alone lifts Llama 3.1 8B on Supreme Court tasks only to 64%, still 16 points below Claude 3.7 Sonnet.
Limitations
- Tasks cover only U.S. Supreme Court and Courts of Appeals documents; generalization to other legal domains or jurisdictions is untested.
- Fine-tuned models remain far from perfect, with high variance in accuracy across tasks and many tasks below intercoder agreement.
- Fine-tuned Lawma models do not generalize well to tasks outside the two databases, so commercial models may be better there.
Experimental setup
Datasets: U.S. Supreme Court Database (SCDB) U.S. Courts of Appeals Database (USCAD) CaselawQA
Benchmarks: CaselawQA
Metrics: accuracy balanced accuracy macro-F1
Baselines: Claude 3.7 Sonnet GPT-4o GPT-4.5 o3-mini Llama 3.1 405B Instruct Llama 3.3 70B Instruct Qwen 2.5 72B Instruct DeepSeek R1 SaulLM 54B Legal-BERT
Builds on: LegalBench MMLU Legal-BERT SaulLM
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.88)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.87)
- Freeing the Law with LOCUS: A Local Ordinance Corpus for the United States (NeurIPS 2026 · similarity 0.85)
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.85)
- Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study (AAAI 2024 · similarity 0.85)
- Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.