Research map / Large language models / Agents, code and math
LEXam: Benchmarking Legal Reasoning on 340 Law Exams
Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of answer choices from 4 to 32 on perturbed MCQs (ablation) | Gemini-2.5-Pro · LEXam perturbed MCQs | accuracy: 68.61 -> 35.62 | Hurt |
| increase number of answer choices from 4 to 32 (ablation) | DeepSeek-V3 · LEXam perturbed MCQs | accuracy: 58.57 -> 16.03 | Hurt |
| increase number of answer choices from 4 to 32 (ablation) | GPT-4o · LEXam perturbed MCQs | accuracy: 53.73 -> 21.81 | Hurt |
| use reasoning models instead of non-reasoning models | LLMs · LEXam open questions | judge score: 70.20 (GPT-5) vs 57.50 (GPT-4.1) | Improved |
| phrase MCQ negatively | reasoning models · LEXam MCQ-16 | accuracy | Hurt |
| increase number of statements from 4 to 5 | reasoning models · LEXam MCQ-16 | accuracy | No clear effect |
| evaluate German instead of English questions | LLMs · LEXam open questions | judge score | Hurt |
| use minimum-score ensemble of GPT-4o, Qwen3-32B, DeepSeek-V3 as judge | LLM judge · Alt-test vs 3 legal experts | winning rate: 1.00 | Improved |
| add MCQ answer option 'None of the statements' | LLMs · LEXam MCQs | accuracy | Hurt |
| include longer model answers | LLMs · LEXam open questions | judge score | Hurt |
| increase question length | LLMs · LEXam open questions | judge score | No clear effect |
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.89)
- Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models (AAAI 2024 · similarity 0.88)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.86)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.86)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.85)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
