AI Paper Atlas

Research map / Large language models / Agents, code and math

Benchmarking Vietnamese Legal Knowledge of Large Language Models

Dong Nguyen Tien, Minh-Anh Nguyen, Thanh D Hoang, Nguyen T Ngoc, Quang M Xuan, Phan P Hai, Nguyen T Anh, Binh Vu, Dung Le

NeurIPS 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VietLegal, a cognitively grounded benchmark for evaluating LLMs' Vietnamese legal knowledge in a civil law system.

Problem

Existing legal benchmarks are English-centric and oriented toward common law, leaving civil law systems (which govern most jurisdictions) underevaluated for LLMs.

Approach

A Vietnamese legal benchmark, VietLegal, designed around cognitive grounding to assess LLM legal knowledge in a civil law jurisdiction.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
move from basic recall (Task 1.4) to article prediction from short queries (Task 3.1)best LLMs on VietLegal · VietLegal zero-shotaccuracy: 87.91 -> 59.83Hurt
domain-adapted legal fine-tuning (CMC-Legal-32B) versus general GPT-4oVietnamese legal LLMs · Court decision predictionaccuracy: +6.17 points (90.67 vs GPT-4o)Improved
domain-adapted legal fine-tuning (CMC-Legal-32B) versus general GPT-4oVietnamese legal LLMs · Unfair contract detectionaccuracy: +15.39 points (73.50 vs GPT-4o)Improved
legal-domain pretraining of a 4B model versus much larger general modelsQwen3-4b-legal-pretrain · Article prediction (Task 3.1)accuracy: 43.83% best open-sourceImproved
scale up general model size (Qwen2.5 3B to 72B)Qwen2.5 · Conflict detection (Task 3.4)Y-F1: 0.00 Y-F1 for 72B, 32B, 7BNo clear effect
scale to larger model for bias detectionQwen2.5 · Bias detection (Task 5.1)accuracy: 14B 57.79 (best), larger models lowerNo clear effect
use few-shot prompting without reasoning instead of zero-shot with chain-of-thought (ablation)LLMs on VietLegal · VietLegal accuracy-based tasksaccuracyImproved
add explicit chain-of-thought reasoning (ablation)LLMs on VietLegal · Level 2 Understanding & StructuringaccuracyHurt
add explicit chain-of-thought reasoning (ablation)LLMs on VietLegal · Level 3 Reasoning & InferenceaccuracyMixed
use surface heuristics (longest answer, PhoBERT similarity) instead of LLMs (ablation)VietLegal MCQ · VietLegal Levels 1, 2, 3, 5accuracy: Longest Answer 20.50 vs Random 25.00 at Level 1No clear effect
evaluate with node/edge F1 instead of ROUGE-L for graph structuring (ablation)Task 2.3 · Legal graph structuringNode-F1 vs Edge-F1: CMC-Legal-32B Node-F1 0.0412, Edge-F1 0.8352Mixed

Experimental setup

Benchmarks: VietLegal

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy