Research map / Large language models / Language, safety and interpretability
SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Mixtral-54B and Mixtral-141B to the legal domain via continued pretraining on a 500B+ token legal corpus, then legal-and-general instruction tuning and DPO alignment with synthetic legal preference data, releasing SaulLM-54B/141B under MIT.
Problem
Legal LLMs have been limited to small scale (7-12B) and small training corpora (under 30B tokens); it is unclear how much scaling model and corpus size improves legal task performance.
Approach
Scale domain adaptation of Mixtral MoE models (54B and 141B) through a full pipeline: continued pretraining on over 540B legal tokens, legal plus general instruction fine-tuning with synthetic dialogues, and DPO with synthetic legal preference pairs judged by an LLM.
Key results
- SaulLM models reach state-of-the-art on LegalBench-Instruct among open-source models, outperforming Llama3-70B-Instruct and Mixtral-Instruct, per the abstract.
- Continued pretraining yields roughly +7% improvement on LegalBench-Instruct across IFT and DPO stages.
- SaulLM-141B has only 44B active parameters; the largest models sometimes do not beat the 54B models on some LegalBench tasks (inverse scaling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pretraining on legal-domain data (ablation) | SaulLM models · Legal-MMLU | performance: approximately +7% | Helped |
| Add DPO preference alignment after instruction fine-tuning (ablation) | SaulLM-medium · LegalBench-Instruct | task-level performance | Mixed |
| Add legal data to the instruction fine-tuning and DPO datasets (ablation) | Mixtral-54-IFT+DPO · LegalBench-Instruct | legal performance | Helped |
| Scale from 54B to 141B (ablation) | SaulLM · LegalBench-Instruct | task-level performance: 20% | Mixed |
| Scale from 54B to 141B (ablation) | SaulLM · LegalBench-Instruct — international law | performance | Hurt |
| Continue pretraining for longer (ablation) | SaulLM-54B-base and SaulLM-141B-base · Pretraining loss | normalized log loss | Helped |
Surprising findings
- Inverse scaling: SaulLM-large and Mixtral-141B do not outperform the smaller SaulLM-medium and Mixtral-54B on some LegalBench-Instruct tasks such as international law.
- Repetitive synthetic data from initial instruction fine-tuning harmed performance during continued pretraining, so legal instruction data was used for annealing instead.
Limitations
- Verbose outputs from aligned models cause answer-parsing failures in evaluation, especially on some interpretation tasks.
- Slightly weaker at following generic instructions than Llama3 and Mixtral-Instruct.
- Instruction tuning and alignment recipes of baselines are hard to replicate; training corpus is English-only and mostly public.
Experimental setup
Datasets: FreeLaw MultiLegal Pile SlimPajama LawInstruct UltraChat UltraInteract Dolphin UltraFeedback Orca LegalBench-Instruct MMLU
Benchmarks: LegalBench-Instruct MMLU (international law, professional law, jurisprudence)
Metrics: balanced accuracy
Baselines: GPT-4 Llama3-70B-Instruct Mixtral-54B-Instruct Mixtral-141B-Instruct SaulLM-7B Legal-FLAN-T5
Builds on: Mixtral SaulLM-7B DPO LegalBench-Instruct
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.88)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.87)
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.87)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.86)
- The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.85)
- Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.