Research map / Label-efficient and robust learning / Domain adaptation and generalization
Achieving Domain-Independent Certified Robustness via Knowledge Continuity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes knowledge continuity, a loss-variability-over-hidden-representation measure that yields domain-independent probabilistic certified robustness, and derives a regularizer (KCReg) that improves robustness of BERT, GPT2, and T5 on IMDB against text attacks.
Problem
Certifying robustness of neural networks across discrete/non-metrizable domains (e.g., language) where Lipschitz continuity does not apply, since no natural input/output distance metric exists and Transformers are not Lipschitz.
Approach
Define robustness as stability of the loss with respect to distances in an intermediate learned metric space (k-volatility, the expected loss change per unit hidden-representation distance); bound this in expectation to get probabilistic certificates that depend only on the loss and hidden metric, not on input modality, norms, or distribution.
Key results
- Average k-volatility across hidden layers explains 35% of variance (R^2=0.35) in TextFooler attack success rate; combined with model properties R^2=0.48.
- KCReg raises IMDB accuracy under TextFooler for BERT from 47.9 (base) to 75.1, versus 56.9 for ALUM.
- KCReg trains up to 2x faster than TextFooler adversarial training and 3x faster than ALUM while improving robustness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| knowledge continuity regularization (KCReg) added to loss | BERT, GPT2, T5 · IMDB under TextFooler and BERT-Attack | adversarial accuracy (%): e.g., BERT TF 47.9 -> 75.1; BA 45.2 -> 84.9 | Helped |
| knowledge continuity regularization (KCReg) added to loss | BERT, GPT2, T5 · IMDB clean accuracy | accuracy (%): BERT 93.6 -> 94.8; GPT2 93.6 -> 94.9; T5 93.7 -> 94.9 | No clear effect |
| KCReg vs ALUM adversarial training | BERT, GPT2, T5 · IMDB under TextFooler and BERT-Attack | adversarial accuracy (%): BERT TF 56.9 (ALUM) vs 75.1 (KCReg) | Helped |
| KCReg vs TextFooler adversarial training | GPT2 and T5 · ANLI R1-R3 and IMDB clean | accuracy (%): GPT2 IMDB clean TF 92.0 vs KCReg 94.9; T5 TF 96.8 vs KCReg 94.9 on IMDB | Mixed |
| knowledge continuity (k-volatility averaged across layers) as predictor of adversarial vulnerability | many language models · IMDB TextFooler attack success rate | R^2: R^2=0.35 alone; 0.48 combined with size and family | Helped |
| regularization with knowledge continuity loss term yields faster training | KCReg vs TextFooler and ALUM training · training speed | training speed relative to baselines: up to 2x vs TextFooler, 3x vs ALUM | Helped |
Surprising findings
- Knowledge continuity is shown to be a statistical rather than analytical property, so it can hold for Transformers that are provably not Lipschitz.
Limitations
- Certification bounds are probabilistic and depend on the choice of metric decomposition.
- Universal approximation with finite Hausdorff dimension decompositions is only conjectured, not proven.
Experimental setup
Datasets: IMDB ANLI
Benchmarks: IMDB ANLI IMDB under TextFooler IMDB under BA
Metrics: accuracy attack success rate R^2 of knowledge continuity vs attack success training speed
Baselines: TextFooler adversarial training (TF) ALUM Base
Builds on: Lipschitz continuity TextFooler ALUM virtual adversarial training
Closest papers by meaning
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.88)
- A Recipe for Improved Certifiable Robustness (ICLR 2024 · similarity 0.87)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.86)
- Certified but Private: Scalable Zero-Knowledge Proofs for the Formal Verification of Neural Networks (NeurIPS 2026 · similarity 0.86)
- Certified Robustness Under Bounded Levenshtein Distance (ICLR 2025 · similarity 0.85)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.