Research map / Large language models / Language, safety and interpretability

AIMS.au: A Dataset for the Analysis of Modern Slavery Countermeasures in Corporate Statements

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AIMS.au, a sentence-level annotated dataset of 5,731 Australian modern slavery statements, and benchmarks zero-shot GPT-3.5/GPT-4o/Llama3.2 against fine-tuned DistilBERT, BERT, Llama2-7B and Llama3.2-3B for detecting mandatory-criteria sentences.

Problem

Scrutinizing thousands of annual corporate modern slavery statements to find concrete countermeasures mandated by the Australian Modern Slavery Act, distinguishing substantive disclosures from vague or distracting text, is infeasible manually and lacks large annotated resources.

Approach

A large, expert-designed sentence-level annotation of modern slavery statements mapped to eleven questions derived from the Act's seven mandatory criteria, with multi-tier annotation quality (hired double annotation plus expert gold sets), and benchmarks showing fine-tuned small models beat zero-shot LLMs.

Key results

• Fine-tuned Llama3.2-3B with context reaches 0.721 macro F1, versus 0.156 for zero-shot Llama3.2-3B without context.
• Zero-shot GPT-4o with context reaches 0.600 macro F1, below fine-tuned models (0.670–0.721 no-context/with-context) on the eleven-question task.
• Zero-shot GPT-3.5 Turbo collapses to 0.130 macro F1 with context, as it predicts nearly all sentences as relevant.

Measured effects reported in the paper

Surprising findings

• GPT-3.5 Turbo performance drops sharply when context is added (0.386 to 0.130 macro F1), defaulting to predicting almost every sentence as relevant.
• Llama2-7B fine-tuned performs worst among fine-tuned models despite larger size, speculated to need more fine-tuning iterations.

Limitations

• Reliance on external annotators may introduce label noise, mostly omissions.
• Figures and tables cannot be analyzed without OCR or vision models, so some context is missing.
• Past and future disclosures are not differentiated by reporting period.

Experimental setup

Datasets: AIMS.au

Benchmarks: AIMS.au gold validation and test sets

Metrics: F1 Macro F1

Baselines: GPT-3.5 Turbo GPT-4o Llama3.2-3B zero-shot DistilBERT BERT Llama2-7B

Builds on: LegalBench LoRA DistilBERT

Closest papers by meaning

• BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text (NeurIPS 2024 · similarity 0.83)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (ICLR 2024 · similarity 0.83)
• Lawma: The Power of Specialization for Legal Annotation (ICLR 2025 · similarity 0.82)
• The SEC Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data (NeurIPS 2026 · similarity 0.82)
• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (NeurIPS 2025 · similarity 0.81)
• SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain (NeurIPS 2024 · similarity 0.81)