Research map / Adversarial robustness and security / Attacks on text and graphs
InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add both InfoBERT regularizers during standard fine-tuning | RoBERTa and BERT · ANLI test, benign training | robust accuracy: 33.2 -> 36.2; 26.5 -> 28.7 | Helped |
| Add both InfoBERT regularizers to adversarial training | RoBERTa · ANLI test, adversarial and benign training | robust accuracy: 57.1 -> 58.3 | Helped |
| Add both InfoBERT regularizers to adversarial training | BERT · ANLI test, adversarial and benign training | robust accuracy: 50.1 -> 51.2 | Helped |
| Add InfoBERT regularizers instead of standard training | BERT · TextFooler adv-SNLI, BERT attack | robust accuracy: 0.0 -> 36.7 | Helped |
| Use both regularizers rather than IBR only (ablation) | BERT · ANLI, standard training | robust accuracy: 27.7 -> 29.2 | Helped |
| Use both regularizers rather than AFR only (ablation) | BERT · ANLI, standard training | robust accuracy: 28.0 -> 29.2 | Helped |
| Use both regularizers rather than IBR only (ablation) | BERT · ANLI, adversarial training | robust accuracy: 29.3 -> 30.9 | Helped |
| Use both regularizers rather than AFR only (ablation) | BERT · ANLI, adversarial training | robust accuracy: 30.3 -> 30.9 | Helped |
| Model p(t|x) with the BERT encoder rather than an auxiliary network (ablation) | BERT Information Bottleneck regularizer · ANLI, standard training | robust accuracy: 27.1 -> 27.7 | Helped |
| Apply the Information Bottleneck regularizer to local rather than global features (ablation) | RoBERTa · ANLI, standard training | robust accuracy: 33.8 -> 33.9 | No clear effect |
| Apply the Information Bottleneck regularizer to local rather than global features (ablation) | RoBERTa · ANLI, standard and adversarial training | robust accuracy: 55.1 -> 56.2 | Helped |
| Increase the Information Bottleneck coefficient beyond the selected value (ablation) | InfoBERT on QA · SQuAD benign and adversarial data | F1 | Hurt |
Closest papers by meaning
- DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION (ICLR 2021 · similarity 0.87)
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (AAAI 2020 · similarity 0.86)
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding (ICLR 2020 · similarity 0.86)
- The MultiBERTs: BERT Reproductions for Robustness Analysis (ICLR 2022 · similarity 0.85)
- $\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts (ICML 2024 · similarity 0.84)
- Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.