Research map / Large language models / Agents, code and math
Can an LLM Reason Like a Lawyer? Benchmarking the ability of LLMs to map the facts of a case to the elements of the applicable legal rule
Shounak Paul, Seungeon Lee, Christoph Engel, Krishna Gummadi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a benchmark that tests whether LLMs map case facts to the elements of applicable legal rules, built from 375 judgments.
Problem
Legal judgment prediction work evaluates whether models reach correct conclusions, not whether their reasoning maps case facts to the elements of a legal provision.
Approach
A benchmark that scores the fact-to-element mapping step of legal reasoning, rather than only final verdicts, built on 375 judgments.
Closest papers by meaning
- LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis (AAAI 2026 · similarity 0.85)
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.85)
- Argumentative Large Language Models for Explainable and Contestable Claim Verification (AAAI 2025 · similarity 0.84)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.84)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.84)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
