Research map / Large language models / AI and society
Evaluation and Incident Prevention in an Enterprise AI Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents a framework for monitoring, benchmarking, and continually improving a deployed enterprise compound AI assistant, centered on a severity-based error taxonomy, coreset-based annotation sampling, and shared development/holdout evaluation sets.
Problem
Monitoring and improving accuracy of multi-component enterprise AI assistants in production, where errors can be incidents, as annotation volume grows and multiple teams develop components.
Approach
Decomposes subjective error-severity judgments (Sev-0/1/2) into less subjective human annotation questions, and uses covariate-aware coreset subsampling (GIGA over LLM2vec embeddings) to reduce required annotations versus uniform sampling, combined with quarterly shared dev/holdout sets for end-to-end regression and impact forecasting.
Key results
- Coreset sampling with GIGA reduces the number of annotations needed to match uniform-sampling RMSE by roughly 20-30% across coreset sizes 100-450.
- For example, a 100-sample coreset matches the RMSE of 125 uniform samples (20% reduction); a 450-sample coreset matches 640 (30%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 100 | samples needed for equivalent RMSE: 100 vs 125; 20% | Helped |
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 200 | samples needed for equivalent RMSE: 200 vs 275; 27% | Helped |
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 300 | samples needed for equivalent RMSE: 300 vs 415; 28% | Helped |
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 350 | samples needed for equivalent RMSE: 350 vs 480; 27% | Helped |
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 400 | samples needed for equivalent RMSE: 400 vs 555; 28% | Helped |
| Use GIGA coreset sampling instead of uniform sampling | Production-interaction annotation sampling · Annotation sampling, coreset size 450 | samples needed for equivalent RMSE: 450 vs 640; 30% | Helped |
Limitations
- Severity labels rely on subjective human annotation, with expert adjudication of disagreements.
- Linear-function assumption over fixed embeddings for quality modeling.
- Evaluation of the sampling method is limited to a single production system and does not report end-to-end impact validation in detail.
Experimental setup
Datasets: production query/answer traffic with human annotations
Metrics: root mean squared error of error-category proportions percentage reduction in required samples
Baselines: uniform random sampling
Builds on: GIGA LLM2vec RAG
Closest papers by meaning
- Lessons for Editors of AI Incidents from the AI Incident Database (AAAI 2025 · similarity 0.84)
- EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting (NeurIPS 2026 · similarity 0.83)
- Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.83)
- AI Testing Should Account for Sophisticated Strategic Behaviour (NeurIPS 2025 · similarity 0.83)
- Position: In-House Evaluation Is Not Enough. Towards Robust Third-Party Evaluation and Flaw Disclosure for General-Purpose AI (ICML 2025 · similarity 0.82)
- Position: AI Evaluation Should Learn from How We Test Humans (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.