Research map / Large language models / Language, safety and interpretability
Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes context-aware testing (CAT) and SMART, an LLM-driven tabular model testing system that generates failure hypotheses from context and falsifies them statistically on data.
Problem
Data-only ML testing searches many subgroups, causing high false positive and false negative rates and surfacing practically meaningless slices, because it ignores contextual knowledge about where models are likely to fail.
Approach
Use an LLM to hypothesize relevant failure slices from task context and dataset description, operationalize them as data queries, and keep only those that are statistically supported by data via a self-falsification step, thereby limiting the number of tests and focusing on meaningful failures.
Key results
- On the prostate cancer dataset, SMART's top slice has |Delta Acc| 0.37 ± 0.03 for logistic regression versus 0.26 for the best baseline (Sliceline) and 0.17 for the no-self-falsification ablation.
- SMART reports the lowest false negative rates in the simulated setting, e.g. FNR 0.00 with one corrupted subgroup versus 0.05 for DivExplorer and 0.25 for Sliceline.
- SMART's share of irrelevant synthetic features in slices stays robust as irrelevant features increase, whereas data-only methods' false positives increase.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMART instead of Sliceline to identify the top failure slice | Prostate cancer testing with LogisticRegression · Prostate cancer test data | |Δ Acc|: 0.26 ± 0.06 -> 0.37 ± 0.03 | Helped |
| Use SMART instead of Sliceline to identify the top failure slice | Prostate cancer testing with SVM · Prostate cancer test data | |Δ Acc|: 0.26 ± 0.06 -> 0.37 ± 0.03 | Helped |
| Use SMART instead of Sliceline to identify the top failure slice | Prostate cancer testing with XGBClassifier · Prostate cancer test data | |Δ Acc|: 0.18 ± 0.09 -> 0.26 ± 0.06 | Helped |
| Use SMART instead of Sliceline to identify the top failure slice | Prostate cancer testing with MLPClassifier · Prostate cancer test data | |Δ Acc|: 0.26 ± 0.06 -> 0.37 ± 0.03 | Helped |
| Add self-falsification (ablation) | SMART with LogisticRegression · Prostate cancer test data | |Δ Acc|: 0.17 ± 0.01 -> 0.37 ± 0.03 | Helped |
| Use SMART instead of Divexplorer | Synthetic recidivism testing with one corrupted subgroup · Synthetic recidivism, n=1 corrupted subgroup | FNR: 0.05 ± 0.22 -> 0.00 ± 0.00 | Helped |
| Use SMART instead of Divexplorer | Synthetic recidivism testing with two corrupted subgroups · Synthetic recidivism, n=2 corrupted subgroups | FNR: 0.40 ± 0.35 -> 0.05 ± 0.15 | Helped |
| Use SMART instead of pysubgroup beam | Synthetic recidivism testing with three corrupted subgroups · Synthetic recidivism, n=3 corrupted subgroups | FNR: 0.68 ± 0.17 -> 0.38 ± 0.27 | Helped |
Surprising findings
- Data-only methods such as Slicefinder find top slices with near-zero accuracy discrepancy (0.01) on the prostate task, below the no-falsification SMART ablation.
Limitations
- Limited to tabular data with interpretable feature names; extension to images or text is not addressed.
- LLM biases cannot be guaranteed absent, though self-falsification mitigates them.
- Requires sufficient real data for falsification; small datasets may limit it.
Experimental setup
Datasets: loan breast cancer diabetes COMPAS recidivism OULAD prostate cancer simulated recidivism data
Benchmarks: loan breast cancer diabetes COMPAS recidivism OULAD prostate cancer simulated recidivism
Metrics: |Delta Acc| of top slice false positive proportion of irrelevant features false negative rate proportion of corrupted subgroup identified
Baselines: Autostrat Slicefinder Sliceline pysubgroup DivExplorer SMART_NSF
Builds on: Autostrat SliceFinder Sliceline DivExplorer pysubgroup
Closest papers by meaning
- Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.87)
- Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing (ICML 2025 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.86)
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.86)
- Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models (ICML 2026 · similarity 0.86)
- Let's (not) just put things in Context: Test-time Training for Long-context LLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.