Research map / Learning theory and algorithms / Fairness
CTBench: A Library and Benchmark for Certified Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CTBench, a unified library and benchmark for deterministic certified training (L-infinity) that re-tunes eight methods under a fair schedule, batch-norm fixes, and a common certification method, then analyzes loss fragmentation, shared mistakes, utilization, regularization, and OOD generalization of certified models.
Problem
Certified training methods are evaluated under inconsistent training schedules, certification methods, and under-tuned hyperparameters, so their reported performance is not comparable and the true state of the art is unclear.
Approach
A single codebase implementing all major IBP-based and related certified training algorithms, with fixed batch-norm handling, a common MN-BaB verifier, a shared training schedule, and systematic hyperparameter tuning, which re-establishes stronger SOTA and reveals that claimed advantages of recent methods shrink under fair comparison.</1>
Key results
- On MNIST eps=0.3, MTL-IBP/TAPS reach about 93.9-94.0% certified accuracy in CTBench, surpassing literature numbers for most methods.
- The relative certified error reduction of recent SOTA over IBP shrinks from 8.15% (literature) to 2.48% (CTBench) on MNIST eps=0.3.
- On CIFAR-10 eps=8/255, certified accuracy in CTBench is 0.03% lower than the literature best but natural accuracy is 0.93% higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TAPS instead of IBP training | CNN7 certified training · MNIST ε=0.1 | certified accuracy: 98.26 -> 98.52 | Helped |
| Add TAPS instead of IBP training | CNN7 certified training · MNIST ε=0.3 | certified accuracy: 93.80 -> 93.95 | Helped |
| Use MTL-IBP instead of IBP training | CNN7 certified training · CIFAR-10 ε=2/255 | certified accuracy: 55.99 -> 64.41 | Helped |
| Use MTL-IBP instead of IBP training | CNN7 certified training · CIFAR-10 ε=8/255 | certified accuracy: 35.28 -> 35.41 | Helped |
| Use MTL-IBP instead of IBP training | CNN7 certified training · TinyImageNet ε=1/255 | certified accuracy: 19.82 -> 27.73 | Helped |
| Add L1 regularization with weight 2·10^-6 (ablation) | IBP-trained CNN7 · MNIST 0.1 | certified accuracy: 98.21 -> 98.26 | Helped |
| Add L1 regularization with weight 1·10^-6 (ablation) | IBP-trained CNN7 · MNIST 0.3 | certified accuracy: 93.56 -> 93.82 | Helped |
| Add L1 regularization with weight 1·10^-6 (ablation) | IBP-trained CNN7 · CIFAR-10 2/255 | certified accuracy: 55.45 -> 55.99 | Helped |
| Add L1 regularization with weight 1·10^-6 (ablation) | IBP-trained CNN7 · CIFAR-10 8/255 | certified accuracy: 35.28 -> 34.36 | Hurt |
| Add L1 regularization with weight 2·10^-6 (ablation) | IBP-trained CNN7 · TinyImageNet 1/255 | certified accuracy: 19.04 -> 19.82 | Helped |
| Reduce attack steps to 1 for MTL-IBP (ablation) | MTL-IBP · certified training settings | certified accuracy | Helped |
| Reduce regularization strength (ablation) | certified training · small perturbation radii | certified accuracy | Helped |
Surprising findings
- Adversarial and certified training both reduce loss fragmentation, and advanced certified methods deactivate more neurons than IBP on MNIST eps=0.1, which was previously thought detrimental.
- Certified training gives better adversarial accuracy than PGD training at large epsilon (e.g., 94.66% vs 93.83% on MNIST eps=0.3).
Limitations
- Only deterministic L-infinity certified training is covered; randomized certified training and other norms are excluded.
- Only adversarial robustness and a few corruption types are considered; patch robustness is not studied.
- Literature results are compared under differing verifiers and budgets, so the comparison depends on the chosen CTBench settings.
Experimental setup
Datasets: MNIST CIFAR-10 TinyImageNet MNIST-C CIFAR-10-C
Benchmarks: MNIST CIFAR-10 TinyImageNet MNIST-C
Metrics: certified accuracy natural accuracy adversarial accuracy certified error propagation tightness unstable neuron ratio model utilization
Baselines: PGD training EDAC IBP CROWN-IBP SABR TAPS STAPS MTL-IBP Adaptive Randomised Smoothing
Builds on: IBP CROWN-IBP SABR TAPS MTL-IBP
Closest papers by meaning
- Halt Fast! Early Stopping for Certified Robustness (NeurIPS 2026 · similarity 0.86)
- Learning Better Certified Models from Empirically-Robust Teachers (NeurIPS 2026 · similarity 0.85)
- Understanding Certified Training with Interval Bound Propagation (ICLR 2024 · similarity 0.85)
- Sparse Transfer Learning Accelerates and Enhances Certified Robustness: A Comprehensive Study (AAAI 2025 · similarity 0.85)
- A Recipe for Improved Certifiable Robustness (ICLR 2024 · similarity 0.84)
- Benchmarking Predictive Coding Networks -- Made Simple (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.