Research map / Large language models / Agents, code and math
Shh, don't say that! Domain Certification in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces domain certification for LLMs and VALID, a rejection-sampling method that uses a small in-domain guide model to give provable upper bounds on the probability of out-of-domain outputs under adversarial prompts.
Problem
LLMs deployed for narrow domains can be adversarially prompted to produce out-of-domain outputs, and existing guardrails offer no provable guarantees against jailbreaks.
Approach
Rejection-sample responses from a general LLM, accepting only those whose length-normalized log-likelihood ratio against a small in-domain guide model is bounded by k; this yields a provable adversarial upper bound 2^{kN} T G(y) on the probability of any out-of-domain output, independent of the prompt.
Key results
- Unconstrained model scores 73.4% on PubMedQA; certifying at epsilon=1e-5 keeps 66.7% (-6.7%), and at epsilon=1e-10 gives 47.7% (-25.7%).
- 95% of OOD samples have atomic certificates below 1e-10 across all three setups.
- On MedicalQA, median log10 constriction ratio of 19.00 at Youden's J optimum (k=5.35, TRR 0.99, FRR 0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VALID with an ε=10^-5 certificate | Llama-3-8B · PubMedQA | certified score: 73.4% -> 66.7% (-6.7%) | Hurt |
| Tighten the VALID certificate from ε=10^-5 to ε=10^-10 | Llama-3-8B with VALID · PubMedQA | certified score: 66.7% -> 47.7% | Hurt |
| Use VALID rejection sampling instead of the unconstrained model | Llama-3-8B · MedicalQA OOD samples | constriction ratio: 24.23 | Helped |
| Set k=5.35 for VALID | Llama-3-8B with VALID · MedicalQA OOD classification | true rejection rate: 0.99 | Helped |
| Set k=5.35 for VALID | Llama-3-8B with VALID · MedicalQA in-domain samples | false rejection rate: 0.01 | Helped |
| Use a smaller rejection threshold k | VALID · MedicalQA | certificate tightness | Helped |
| Increase the number of rejection attempts T | VALID · MedicalQA in-domain samples | acceptance rate | Helped |
| Increase the number of rejection attempts T | VALID · MedicalQA OOD certification | domain-certificate tightness: linearly | Hurt |
| Use guide model G alone instead of combining it with L through VALID (ablation) | MedicalQA · MedicalQA | task performance | Hurt |
| Adjust for the short-sequence confounder in the OOD dataset (ablation) | VALID · MedicalQA | certificate disentanglement | Helped |
Surprising findings
- Certificates on out-of-domain samples are significantly tighter than on in-domain samples, meaning the guide model rejects OOD responses strongly even with limited language ability.
Limitations
- Guide model G lacks context, so marginally in-domain responses that are out-of-domain conditionally on the prompt may be accepted.
- Guide model has limited semantic understanding and may not align the model, only shush it.
- Adversaries may try to copy prompt tokens into G, and certificates depend on the quality of the guide model.
Experimental setup
Datasets: TinyShakespeare IMDB RTE SST2 Bible 20 Newsgroups PubMedQA SQuAD CharTask
Benchmarks: PubMedQA PubMedQA@epsilon
Metrics: false rejection rate true rejection rate atomic certificate epsilon constriction ratio PubMedQA accuracy under certificate
Baselines: unconstrained Gemma-2-2b / Llama-3-8B model L guide model G alone
Builds on: Renyi divergence likelihood ratio test for OOD detection PubMedQA
Closest papers by meaning
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.86)
- RvLLM: LLM Runtime Verification with Domain Knowledge (NeurIPS 2025 · similarity 0.85)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.85)
- Position: TrustLLM: Trustworthiness in Large Language Models (ICML 2024 · similarity 0.85)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.85)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.