Research map / Large language models / Alignment and preferences
Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring
Wenhao Xu, Akshatha Arodi, Jian-Yun Nie, Arsène Fansi Tchango
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CA-Judge rule-alignment reward on top of format and correctness rewards (ablation) | CALLM (Qwen2.5-3B) · AIMS.au macro F1 | F1: 0.559 -> 0.639 | Improved |
| add CA-Judge reward (ablation) | CALLM (Qwen2.5-3B) · AIMS.au C3 Risk Description | F1: 0.564 -> 0.712 | Improved |
| add CA-Judge reward (ablation) | CALLM (Qwen2.5-3B) · AIMS.au C4 Remediation | F1: 0.397 -> 0.570 | Improved |
| add CA-Judge reward (ablation) | CALLM (Qwen2.5-3B) · AIMS.au C5 Effectiveness | F1: 0.394 -> 0.439 | Mixed |
| add CA-Judge reward (ablation) | CALLM (Qwen2.5-3B) · AIMS.au C2 Structure | F1: 0.535 -> 0.572 | Improved |
| fine-tune with format and correctness rewards (GRPO) vs pre-trained base (ablation) | Qwen2.5-3B-Instruct · AIMS.au macro F1 | F1: 0.293 -> 0.559 | Improved |
| add few-shot examples to CoT prompt | GPT-4o · AIMS.au macro F1 | F1: 0.559 -> 0.617 | Improved |
| CALLM vs GPT-4o on easy Approval criterion | CALLM · AIMS.au Approval | F1: 0.855 -> 0.786 | Hurt |
| train on AU, test cross-jurisdiction on UK | CALLM · UK held-out | macro F1: 0.560 -> 0.620 | Improved |
| train on AU, test cross-jurisdiction on Canada | CALLM · Canada held-out | macro F1: 0.589 -> 0.617 | Improved |
| CA-Judge-trained CALLM vs GPT-4o few-shot on rule alignment | CALLM · AIMS.au test | CA-Judge score: 0.53 -> 0.74 | Improved |
| raise GRPO KL weight beta from 0.001 to 0.04 (ablation) | GRPO training · completion length during training | reward hacking (long completions) | Improved |
Closest papers by meaning
- AIMS.au: A Dataset for the Analysis of Modern Slavery Countermeasures in Corporate Statements (ICLR 2025 · similarity 0.89)
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.83)
- Position: Contextual Integrity is Inadequately Applied to Language Models (ICML 2025 · similarity 0.83)
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.83)
- CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.83)
- Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
