Research map / Large language models / Agents, code and math
Benchmarking Vietnamese Legal Knowledge of Large Language Models
Dong Nguyen Tien, Minh-Anh Nguyen, Thanh D Hoang, Nguyen T Ngoc, Quang M Xuan, Phan P Hai, Nguyen T Anh, Binh Vu, Dung Le
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VietLegal, a cognitively grounded benchmark for evaluating LLMs' Vietnamese legal knowledge in a civil law system.
Problem
Existing legal benchmarks are English-centric and oriented toward common law, leaving civil law systems (which govern most jurisdictions) underevaluated for LLMs.
Approach
A Vietnamese legal benchmark, VietLegal, designed around cognitive grounding to assess LLM legal knowledge in a civil law jurisdiction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from basic recall (Task 1.4) to article prediction from short queries (Task 3.1) | best LLMs on VietLegal · VietLegal zero-shot | accuracy: 87.91 -> 59.83 | Hurt |
| domain-adapted legal fine-tuning (CMC-Legal-32B) versus general GPT-4o | Vietnamese legal LLMs · Court decision prediction | accuracy: +6.17 points (90.67 vs GPT-4o) | Improved |
| domain-adapted legal fine-tuning (CMC-Legal-32B) versus general GPT-4o | Vietnamese legal LLMs · Unfair contract detection | accuracy: +15.39 points (73.50 vs GPT-4o) | Improved |
| legal-domain pretraining of a 4B model versus much larger general models | Qwen3-4b-legal-pretrain · Article prediction (Task 3.1) | accuracy: 43.83% best open-source | Improved |
| scale up general model size (Qwen2.5 3B to 72B) | Qwen2.5 · Conflict detection (Task 3.4) | Y-F1: 0.00 Y-F1 for 72B, 32B, 7B | No clear effect |
| scale to larger model for bias detection | Qwen2.5 · Bias detection (Task 5.1) | accuracy: 14B 57.79 (best), larger models lower | No clear effect |
| use few-shot prompting without reasoning instead of zero-shot with chain-of-thought (ablation) | LLMs on VietLegal · VietLegal accuracy-based tasks | accuracy | Improved |
| add explicit chain-of-thought reasoning (ablation) | LLMs on VietLegal · Level 2 Understanding & Structuring | accuracy | Hurt |
| add explicit chain-of-thought reasoning (ablation) | LLMs on VietLegal · Level 3 Reasoning & Inference | accuracy | Mixed |
| use surface heuristics (longest answer, PhoBERT similarity) instead of LLMs (ablation) | VietLegal MCQ · VietLegal Levels 1, 2, 3, 5 | accuracy: Longest Answer 20.50 vs Random 25.00 at Level 1 | No clear effect |
| evaluate with node/edge F1 instead of ROUGE-L for graph structuring (ablation) | Task 2.3 · Legal graph structuring | Node-F1 vs Edge-F1: CMC-Legal-32B Node-F1 0.0412, Edge-F1 0.8352 | Mixed |
Experimental setup
Benchmarks: VietLegal
Closest papers by meaning
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.90)
- Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models (AAAI 2024 · similarity 0.87)
- LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis (AAAI 2026 · similarity 0.86)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.85)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.85)
- J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
