Research map / Large language models / Language, safety and interpretability
Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CASAM, a causality-aware self-attention over OIE-derived graphs, plus eight legal-specific adversarial attacks for model selection, evaluated on legal judgment prediction benchmarks built on Legal-BERT.
Problem
Pretrained language models for legal text prediction learn spurious correlations instead of causal rationales, and standard i.i.d. train/test evaluation with SOTA-chasing fails to select causality-based models.
Approach
Restricts attention interactions between causal and non-causal words, using an adjacency matrix from open information extraction graphs to reweight self-attention (S' = αS + (1−α)S⊙A), and selects models using causality-invariant legal adversarial attacks rather than only i.i.d. validation data.
Key results
- CASAM reaches 73.8 micro-F1 / 68.5 macro-F1 on ECtHR Task A, versus 72.3 / 66.0 for Legal-BERT.
- CASAM achieves certified ratios above 99% on the three datasets under the eight legal-specific attacks (except some LEDGAR word-level cases, 96.2–98.9%).
- On LEDGAR, CASAM reaches 88.7 micro-F1 and 83.5 macro-F1, versus 88.2 / 81.9 for Legal-BERT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CASAM causal attention | Legal-BERT · ECtHR Task A | µ-F1: 72.3 -> 73.8 | Helped |
| add CASAM causal attention | Legal-BERT · ECtHR Task A | m-F1: 66.0 -> 68.5 | Helped |
| add CASAM causal attention | Legal-BERT · ECtHR Task B | µ-F1: 80.6 -> 81.4 | Helped |
| add CASAM causal attention | Legal-BERT · ECtHR Task B | m-F1: 75.2 -> 76.0 | Helped |
| add CASAM causal attention | Legal-BERT · LEDGAR | µ-F1: 88.2 -> 88.7 | Helped |
| add CASAM causal attention | Legal-BERT · LEDGAR | m-F1: 81.9 -> 83.5 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on ECtHR Task A · ECtHR Task A | µ-F1: 72.3 -> 73.8 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on ECtHR Task A · ECtHR Task A | m-F1: 66.0 -> 68.5 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on ECtHR Task B · ECtHR Task B | µ-F1: 80.6 -> 81.4 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on ECtHR Task B · ECtHR Task B | m-F1: 75.2 -> 76.0 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on LEDGAR · LEDGAR | µ-F1: 88.2 -> 88.7 | Helped |
| add causal attention (rather than removing it) (ablation) | CASAM on LEDGAR · LEDGAR | m-F1: 81.9 -> 83.5 | Helped |
Surprising findings
- Predictions of Legal-BERT on ECtHR can be reversed by masking a single punctuation mark or character, and about 5% of words reserve 92.13% of its performance.
- Legal-BERT's top evidence words are dominated by punctuation and function words rather than content words.
Limitations
- Performance depends on the accuracy of the OIE tool, which cannot precisely separate causal from non-causal information.
- Word-level attacks on LEDGAR yield notably weaker robustness, since topic-category classification relies on content words that attacks mask.
- Evaluated only with Legal-BERT as backbone; generalization to other backbones is left to future work.
Experimental setup
Datasets: ECtHR Task A ECtHR Task B LEDGAR
Benchmarks: ECtHR Task A ECtHR Task B LEDGAR
Metrics: micro-F1 macro-F1 certified ratio
Baselines: TFIDF+SVM BERT RoBERTa DeBERTa Longformer BigBird CaseLaw-BERT Legal-BERT
Builds on: Legal-BERT Transformer self-attention LexGLUE BERT-ATTACK
Closest papers by meaning
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.90)
- Causality can systematically address the monsters under the bench(marks) (NeurIPS 2026 · similarity 0.88)
- Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors (ICML 2025 · similarity 0.87)
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.87)
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.