Research map / Label-efficient and robust learning / Recognition and long-tail
DELTA: Pre-Train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a BERT-based dual encoder for legal case retrieval by using a deep decoder's cross-attention word alignment between Fact and Reasoning sections to find key facts, then contrastively pulling the [CLS] embedding toward key facts and away from non-key facts.
Problem
Pre-trained encoders for legal case retrieval optimize textual semantic similarity via [CLS], but legal relevance depends on key facts that determine judgment, and lengthy cases contain many non-key facts.
Approach
Uses unsupervised word alignment (cross-attention of a six-layer deep decoder trained to translate Fact into Reasoning) to label key vs. non-key fact tokens, then applies a contrastive loss so the case [CLS] embedding is close to key-fact representations and far from non-key ones, combined with shallow-decoder MLM-style reconstruction bottlenecks (following SAILER).
Key results
- Zero-shot on LeCaRD, DELTA reaches F1 0.3236 and NDCG@10 0.8117, above SAILER's 0.3142 and 0.7979.
- Zero-shot on COLIEE2023, DELTA reaches F1 0.1075 and R@500 0.8956 versus SAILER's F1 0.0831 and R@500 0.8391.
- With fine-tuning on COLIEE2022, DELTA reaches F1 0.2938 versus SAILER's 0.2757.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DELTA instead of SAILER | legal case retrieval · LeCaRD zero-shot | F1_score: 0.3142 -> 0.3236 | Helped |
| Use DELTA instead of SAILER | legal case retrieval · CAIL2022-LCR zero-shot | F1_score: 0.3671 -> 0.3767 | Helped |
| Use DELTA instead of the strongest baseline | legal case retrieval · COLIEE2022 zero-shot | F1_score: 0.0957 -> 0.1064 | Helped |
| Use DELTA instead of the strongest baseline | legal case retrieval · COLIEE2023 zero-shot | F1_score: 0.0969 -> 0.1075 | Helped |
| Use DELTA instead of SAILER | legal case retrieval · COLIEE2022 fine-tuned | F1_score: 0.2757 -> 0.2938 | Helped |
| Use DELTA instead of SAILER | legal case retrieval · COLIEE2023 fine-tuned | F1_score: 0.1629 -> 0.1711 | Helped |
| Add L_CON contrastive learning (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.0831 -> 0.1075 | Helped |
| Add L_TLM structural word-alignment objective (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.0725 -> 0.1075 | Helped |
| Add the reasoning shallow-decoder objective L_REA (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.0823 -> 0.1075 | Helped |
| Add the decision shallow-decoder objective L_DEC (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.0790 -> 0.1075 | Helped |
| Add the MLM objective L_MLM (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.1035 -> 0.1075 | Helped |
| Use 6 rather than 2 deep decoder layers (ablation) | DELTA · COLIEE2023 zero-shot | F1_score: 0.0839 -> 0.1075 | Helped |
Surprising findings
- On the Chinese benchmarks DELTA shows little improvement over SAILER, which the authors attribute to the Chinese system treating cases with similar causes as relevant.
- Removing the MLM loss slightly improves COLIEE2023 zero-shot F1 (0.1035 vs 0.1075 for full model is lower, but ablation shows w/o MLM precision 0.0796 vs 0.0828), so MLM contributes less than the alignment and contrastive losses.
Limitations
- Gains on Chinese benchmarks are marginal relative to SAILER.
- Key-fact selection relies on a fixed top-p% ratio (60%) tuned by grid search.
- Evaluated only on four benchmarks with limited query counts.
Experimental setup
Datasets: LeCaRD CAIL2022-LCR COLIEE2022 COLIEE2023 China Judgment Online corpus case.law US court corpus
Benchmarks: LeCaRD CAIL2022-LCR COLIEE2022 COLIEE2023
Metrics: Precision@5 Recall@5 F1 NDCG@10 NDCG@30 R@100 R@500 R@200
Baselines: BM25 QLD BERT RoBERTa Condenser coCondenser SEED COT-MAE RetroMAE SAILER Lawformer LEGAL-BERT
Builds on: SAILER RetroMAE Garg et al. multitask word alignment BERT
Closest papers by meaning
- Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models (AAAI 2024 · similarity 0.86)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.84)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.83)
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.82)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.82)
- BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.