Research map / Large language models / Agents, code and math
LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LexEval, a 23-task, 14,150-question Chinese legal benchmark organized by a six-level legal cognitive ability taxonomy, and evaluates 38 general and legal-specific LLMs in zero- and few-shot settings.
Problem
Existing general and NLP-derived benchmarks do not comprehensively assess LLM capabilities in the Chinese legal domain, including legal knowledge application and legal ethics.
Approach
A Legal Cognitive Ability Taxonomy (LexAbility: Memorization, Understanding, Logic Inference, Discrimination, Generation, Ethic) organizes 23 tasks built from existing legal datasets, national exam questions, and expert-annotated data, all standardized to multiple-choice (accuracy) or ROUGE-L generation metrics.
Key results
- GPT-4 achieves the best average zero-shot score of 52.4%, ahead of Qwen-14B-Chat at 48.6%.
- Legal-specific LLMs do not consistently outperform general LLMs; e.g., ChatLaw-33B averages 30.0% zero-shot.
- Most models perform poorly on Legal Evolution (1-3) and the Discrimination and Generation levels, with few-shot prompting giving unstable gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase Qwen model size from 7B to 14B | Qwen chat models · LexEval | Average: 44.8 -> 48.6 | Helped |
| Use chat-tuned rather than base Baichuan-13B | Baichuan-13B · LexEval | Average: 31.3 -> 36.4 | Helped |
| Add few-shot examples | GPT-4 · LexEval | Average: 52.4 -> 53.7 | Helped |
| Add few-shot examples | Qwen-14B-Chat · LexEval | Average: 48.6 -> 49.9 | Helped |
| Add few-shot examples | Qwen-7B-Chat · LexEval | Average: 44.8 -> 42.9 | Hurt |
| Use few-shot rather than zero-shot prompting across models | Evaluated LLMs · LexEval | Performance | Mixed |
Surprising findings
- Legal-specific LLMs do not always outperform general LLMs, possibly because continued legal pre-training degrades base model abilities.
- Few-shot prompting yields unstable improvements, sometimes degrading performance (e.g., Qwen-14B-Chat), suggesting in-context learning is not an ideal way to inject legal knowledge.
Limitations
- Tasks mainly cover the Statute Law system and not the Case Law system.
- Rouge-L may not fully capture legal generation quality.
- Some long inputs exceed maximum context lengths of evaluated models and are truncated.
Experimental setup
Datasets: JEC-QA CAIL2018 CAIL2019 CAIL2020 CAIL2021 CAIL2022 LeCaRD National Uniform Legal Profession Qualification Examination
Benchmarks: LexEval
Metrics: Accuracy Rouge-L
Baselines: GPT-4 ChatGPT Qwen-14B-Chat Qwen-7B-Chat ChatLaw-33B Fuzi-Mingcha InternLM-7B-Chat ChatGLM3
Builds on: Bloom's taxonomy CAIL JEC-QA LeCaRD
Closest papers by meaning
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.88)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.88)
- Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models (AAAI 2024 · similarity 0.87)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.87)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.87)
- LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.