Research map / Large language models / Language, safety and interpretability

Statistical Hypothesis Testing for Auditing Robustness in Language Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes distribution-based perturbation analysis (DBPA), a permutation-test hypothesis testing framework that compares Monte Carlo output distributions of black-box LLMs in a cosine-similarity space to audit robustness to input perturbations and model changes.

Problem

Testing whether LLM outputs change under an intervention (input perturbation or model swap) is hard because outputs are stochastic and full output distributions are intractable and not semantically interpretable.

Approach

Reframe LLM perturbation analysis as a frequentist two-sample test: pool k samples from original and perturbed prompts, compare within-group vs cross-group embedding cosine similarity distributions via Jensen-Shannon divergence, and obtain p-values by permutation; the JSD value serves as effect size.

Key results

• In the persona case study, GPT-4 and GPT-3.5 show no significant effect for medical personas while several non-medical personas (e.g., comedian at 0.32***) are significant; smaller models show less consistent patterns.
• TPR/FPR analysis shows the best model depends on the allowed FPR: Phi-3 preferred at 1% FPR, Mistral-7B at 5% FPR.
• Cost analysis estimates embedding costs of $2–$13 for 50,000 perturbations depending on the OpenAI embedding model.

Measured effects reported in the paper

Surprising findings

• Longer prompts (above about 800 tokens) make persona-induced output shifts undetectable, so detectability decreases with input length.
• Effect-size magnitudes and significance patterns vary substantially across embedding models, and BLEU/ROUGE produce noisier results than embedding-based similarity.

Limitations

• Authors describe the work as conceptual with limited empirical evidence; future work should test other domains.
• Results depend on design choices such as embedding function, distance measure, and sample size; these choices lack selection guidelines.
• Non-rejection only means no detectable shift under the chosen similarity function, not identical output distributions.

Experimental setup

Datasets: synthetic patient feature prompts for NICE CVD guidelines

Benchmarks: persona perturbation healthcare prompts CVD treatment recommendation TPR/FPR cross-model alignment vs GPT-4

Metrics: JSD effect size permutation p-value TPR FPR ROC AUC

Baselines: BLEU ROUGE BERTScore MoverScore

Builds on: permutation testing Jensen-Shannon divergence

Closest papers by meaning

• An Auditing Test to Detect Behavioral Shift in Language Models (ICLR 2025 · similarity 0.90)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
• An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.88)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.88)
• Brittlebench: Quantifying LLM robustness via prompt sensitivity (NeurIPS 2026 · similarity 0.88)
• Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.88)