Research map / Large language models / Retrieval and knowledge
Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLeQA, a 1,868-question expert-annotated French statutory-law long-form QA dataset, and evaluates a retrieve-then-read pipeline (CamemBERT bi-encoder retriever plus 7B instruction-tuned LLaMA-based readers) with interpretable paragraph-level rationales.
Problem
Existing legal question answering systems are narrow in domain or give brief answers, leaving laypeople without accessible, informative legal guidance on any statutory law question.
Approach
Combine a siamese bi-encoder retriever over Belgian statutory articles with an LLM reader that generates long-form answers and extractive paragraph-level rationales, adapted via in-context learning or QLoRA, on a new expert-annotated French dataset built from BSARD.
Key results
- Fine-tuned CamemBERT retriever reaches 48.6 R@5 and 60.0 MRR@10 on the LLeQA dev set, versus 17.4 R@5 for BM25.
- Finetuning consistently improves METEOR across generators; WizardLM-7B reaches 20.4 METEOR finetuned versus 12.3 zero-shot.
- Rationale extraction F1 is near zero for most 7B open models, while GPT-4 reaches 65.5 F1 for annotation of paragraph-level rationales.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the CamemBERT siamese bi-encoder on LLeQA instead of using BM25 or mE5 | LLeQA retrieval · LLeQA dev | R@5: 17.4 -> 48.6; 15.4 -> 48.6; 16.5 -> 48.6 | Helped |
| Fine-tune the CamemBERT siamese bi-encoder on LLeQA instead of using BM25 or mE5 | LLeQA retrieval · LLeQA dev | R@10: 22.8 -> 60.6; 21.7 -> 60.6; 26.7 -> 60.6 | Helped |
| Fine-tune the CamemBERT siamese bi-encoder on LLeQA instead of using BM25 or mE5 | LLeQA retrieval · LLeQA dev | MRR@10: 22.0 -> 60.0; 25.8 -> 60.0; 28.3 -> 60.0 | Helped |
| Add QLoRA fine-tuning (ablation) | Vicuna · LLeQA test | METEOR: 11.6 -> 19.7 | Helped |
| Add QLoRA fine-tuning (ablation) | WizardLM · LLeQA test | METEOR: 12.3 -> 20.4 | Helped |
| Add QLoRA fine-tuning (ablation) | TÜLU · LLeQA test | METEOR: 2.9 -> 12.7 | Helped |
| Add QLoRA fine-tuning (ablation) | Guanaco · LLeQA test | METEOR: 11.2 -> 20.1 | Helped |
| Add one-shot or few-shot demonstrations instead of zero-shot prompting (ablation) | Vicuna · LLeQA test | METEOR: 11.6 -> 16.2; 11.6 -> 15.3 | Helped |
| Add one-shot or few-shot demonstrations instead of zero-shot prompting (ablation) | WizardLM · LLeQA test | METEOR: 12.3 -> 15.5; 12.3 -> 16.6 | Helped |
| Add one-shot or few-shot demonstrations instead of zero-shot prompting (ablation) | TÜLU · LLeQA test | METEOR: 2.9 -> 4.6; 2.9 -> 8.5 | Helped |
| Add demonstrations instead of zero-shot prompting (ablation) | Guanaco · LLeQA test | METEOR: 11.2 -> 11.2; 11.2 -> 11.3 | No clear effect |
| Increase demonstrations from one-shot to few-shot (ablation) | LLeQA generators · LLeQA test | METEOR: Vicuna: 16.2 -> 15.3; WizardLM: 15.5 -> 16.6; TÜLU: 4.6 -> 8.5; Guanaco: 11.2 -> 11.3 | No clear effect |
Surprising findings
- Automatic METEOR scores are high-looking for fluent answers, yet qualitative analysis shows frequent fabricated facts and conditions in WizardLM and Vicuna outputs.
- Guanaco performs strongly zero-shot on METEOR but its finetuned rationale extraction is zero, and demonstrations did not consistently help.
Limitations
- Generated answers and rationales remain prone to hallucination of facts and sources.
- Automatic metrics like METEOR may not reflect answer quality; human evaluation deferred to future work.
- Rationale labels for most training data are synthetic from gpt-3.5-turbo (47.5 F1 against expert test labels), and the study is limited to 7B models due to compute.
Experimental setup
Datasets: LLeQA BSARD
Benchmarks: LLeQA
Metrics: R@5 R@10 MRR@10 METEOR F1
Baselines: BM25 mE5-base mE5-large Vicuna-7B WizardLM-7B TÜLU-7B Guanaco-7B
Builds on: BSARD QLoRA BM25 NTK-aware RoPE scaling CamemBERT
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.87)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.86)
- DELTA: Pre-Train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment (AAAI 2025 · similarity 0.86)
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.85)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.85)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.