Research map / Large language models / Agents, code and math
Can LLMs Reason Soundly in Law? Auditing Inference Patterns for Legal Judgment
Lu Chen, Yuxuan Huang, Yixing Li, Dongrui Liu, Qihan Ren, ShuaiZhao, Kun Kuang, Zilong Zheng, Quanshi Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| audit legal LLM judgments by interaction patterns rather than output correctness | Qwen2.5-14B, DeepSeek-R1-Distill-Qwen-14B, SaulLM-7B, BAI-Law-13B · ECtHR, LegalBench, CAIL2018, LeCaRD, LEVEN | s_reliable | Hurt |
| judge defendant using text describing another defendant's actions | SaulLM-7B-Instruct · CAIL2018 Andy assault case | s_reliable: 41.5% reliable | Hurt |
| law-specific fine-tuning (BAI-Law-13B vs SaulLM-7B) on the same case | legal LLMs · CAIL2018 Andy case | s_reliable: 41.5% -> 44.5% | No clear effect |
| replace victim occupation 'a judge' with 'a volunteer' or 'a programmer' (ablation) | SaulLM-7B-Instruct · Robbery case | judgment output / reliable interaction effect: 0.22 -> 0.06; Robbery -> Not mentioned | Hurt |
| use law-related occupations (lawyer, policeman) instead of judge (ablation) | SaulLM-7B-Instruct · Robbery case | judgment output | No clear effect |
| inspect interaction orders for chain-of-thought-like reasoning | evaluated legal LLMs · LegalBench, CAIL2018, LeCaRD, LEVEN | interaction order distribution | Hurt |
Closest papers by meaning
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.90)
- Argumentative Large Language Models for Explainable and Contestable Claim Verification (AAAI 2025 · similarity 0.89)
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.87)
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.87)
- Logical Consistency of Large Language Models in Fact-Checking (ICLR 2025 · similarity 0.87)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
