Research map / Architectures and efficiency / RNNs and normalization
Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a low-rank, iteration-free Block Reflector Orthogonal (BRO) layer and a Logit Annealing loss to build BRONet, a Lipschitz network with state-of-the-art certified L2 robustness on CIFAR-10/100, Tiny-ImageNet and ImageNet.
Problem
Certified robustness of Lipschitz neural networks is limited by expensive orthogonal layers and by limited model capacity that hampers margin maximization in standard losses.
Approach
A block-reflector parameterization W = I - 2V(V^T V)^{-1}V^T gives orthogonal layers from a low-rank unconstrained matrix without iterative approximation; the Logit Annealing loss uses a focal-style (1-p_t)^beta annealing term with temperature and offset so that large-margin points are down-weighted, letting limited-capacity Lipschitz models learn appropriate margins for most points.
Key results
- BRONet-L (+LA) reaches 81.6% clean and 70.6% certified accuracy at 36/255 on CIFAR-10, the best in the table.
- On ImageNet, BRONet (+LA) reaches 49.3% clean and 37.6% certified accuracy at 36/255, versus 47.3% and 35.3% for LiResNet.
- With LipConvNet on CIFAR-100, BRO matches or beats SOC and LOT in certified accuracy at several depth-width settings while using fewer parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Logit Annealing to BRONet-L | BRONet-L · CIFAR-10 | clean accuracy: 81.0 -> 81.6 | Helped |
| Add Logit Annealing to BRONet-L | BRONet-L · CIFAR-10, ε=72/255 | certified accuracy: 57.1 -> 57.2 | Helped |
| Add Logit Annealing to BRONet-L | BRONet-L · CIFAR-100, ε=72/255 | certified accuracy: 28.6 -> 29.1 | Helped |
| Add Logit Annealing to BRONet | BRONet · ImageNet, ε=72/255 | certified accuracy: 25.8 -> 27.9 | Helped |
| Replace LiResNet with BRONet+LA using diffusion augmentation | ImageNet (+EDM2 2M) · ImageNet (+EDM2 2M), ε=72/255 | certified accuracy: +2.7 | Helped |
| Replace Lipschitz-regularized convolutional backbone with BRO layers | LiResNet with LA and EDM augmentation · CIFAR-10 (+EDM), ε=72/255 | certified accuracy: 67.0 -> 67.4 | Helped |
| Replace Lipschitz-regularized convolutional backbone with BRO layers | LiResNet with LA and EDM augmentation · CIFAR-100 (+EDM), ε=36/255 | certified accuracy: 48.9 -> 49.1 | Helped |
| Use BRO instead of SOC | LipConvNet (10–48) · CIFAR-100, ε=36/255 | certified accuracy: 34.9 -> 36.2 | Helped |
| Use BRO instead of SOC | LipConvNet (10–48) · Tiny-ImageNet, ε=108/255 | certified accuracy: 11.2 -> 12.3 | Helped |
| Add BRO layers (ablation) | ImageNet model with LA and BRONet architecture · ImageNet, ε=72/255 | certified accuracy: 26.8 -> 27.9 | Helped |
| Use the BRONet architecture (ablation) | ImageNet model with LA and BRO · ImageNet, ε=72/255 | certified accuracy: 26.7 -> 26.8 | Helped |
| Add Logit Annealing (ablation) | ImageNet model with BRONet architecture and BRO · ImageNet, ε=72/255 | certified accuracy: 25.1 -> 27.9 | Helped |
Surprising findings
- The Kaiming-initialized LOT layer can produce a non-orthogonal layer due to Newton-method instability, even with 64-bit precision.
- Adding LA loss slightly lowers clean accuracy on CIFAR-10 with BRONet-M (80.5 to 81.2 is an increase, but LiResNet+LA with diffusion data drops clean accuracy by 0.3).
Limitations
- A single BRO layer is not a universal approximator of orthogonal matrices (low-rank parameterization).
- Some FFT-based orthogonal layers require fewer backbone layers to fit in memory, so comparisons are constrained.
- Gains over LiResNet on CIFAR-10/100 are modest, and some results rely on diffusion-generated extra data.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Metrics: clean accuracy certified accuracy at l2 radius 36/255, 72/255, 108/255
Baselines: Cayley SOC LOT Cholesky SLL CPL AOL Sandwich LiResNet LipConvNet
Builds on: LiResNet SOC LOT CRC certificate regularization Focal loss
Closest papers by meaning
- A Recipe for Improved Certifiable Robustness (ICLR 2024 · similarity 0.92)
- 1-Lipschitz Layers Compared: Memory Speed and Certifiable Robustness (CVPR 2024 · similarity 0.88)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.88)
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.87)
- Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks (ICML 2025 · similarity 0.87)
- A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.