Research map / Large language models / AI and society
AI Evaluation Authorities: A Case Study Mapping Model Audits to Persistent Standards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "Evaluation Authorities," a secured, open-source programmatic evaluation infrastructure that scales a prior NER multilingual-robustness audit into reusable, protected tests that report aggregated findings without exposing data.
Problem
Intelligent system audits are labor-intensive, performed once, and discarded; NER-related AI incidents recur because there is no reproducible, scalable, and secure way to test deployed models for such failure modes.
Approach
Operationalize one-off audits as protected, reusable tests run inside a trusted authority (held-out datasets never exposed to system owners, guarding against Goodhart's Law), producing aggregated reports and relative benchmarks across models and product categories.
Key results
- Demonstrates an NER Evaluation Authority report comparing three HuggingFace NER models on multilingual robustness, with results reported in aggregate without exposing the protected dataset.
- Shows the infrastructure is model-agnostic by evaluating six ResNet depths on image common-corruption blur degradation.
- Argues the framework could identify upstream issues related to three AIID incidents (317, 363, 392) involving likely NER failures on Facebook, though no quantitative incident-prevention result is reported.
Limitations
- Protected test data is not released, so the NER assessment cannot be independently verified by outsiders.
- Evaluation covers a narrow scope (multilingual person-name robustness) and does not directly demonstrate prevention of the cited incidents.
- No defensible threshold of 'good enough' is provided for most assurance metrics.
Experimental setup
Datasets: multilingual person names dataset (protected) ImageNet common corruptions (Hendrycks and Dietterich 2018) AI Incident Database
Benchmarks: multilingual robustness NER assessment common image corruptions robustness test
Metrics: performance difference across person-name sets by language degradation under blur intensity
Baselines: Davlan/xlm-roberta-base-ner-hrl dslim/bert-base-NER Jean-Baptiste/camembert-ner
Builds on: Calix et al. 2022 multilingual NER assessment BentoML Apache Arrow Kubernetes
Closest papers by meaning
- Robust ML Auditing using Prior Knowledge (ICML 2025 · similarity 0.82)
- Evaluation and Incident Prevention in an Enterprise AI Assistant (AAAI 2025 · similarity 0.82)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.82)
- Adaptive auditing of AI systems with anytime-valid guarantees (NeurIPS 2026 · similarity 0.81)
- Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases (NeurIPS 2026 · similarity 0.81)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.