Research map / Architectures and efficiency / CNN design
An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AOC, an explicit orthogonal convolution construction that fuses BCOP and RKO kernels via a fast block-convolution, supporting stride, transposed, grouped and dilated convolutions at low overhead.
Problem
Orthogonal convolutions are useful for Lipschitz/robust networks, normalizing flows, GANs and RNNs, but existing constructions lack support for strides, transposed convolutions, groups and dilation, and are costly to scale to large images and models.
Approach
Compose a BCOP kernel (arbitrary size, stride 1) with an RKO kernel (kernel size equal to stride) via the block-convolution operator, with a proven channel-size choice, yielding strictly orthogonal strided convolutions whose cost is independent of input resolution; also derives native transposed, grouped and dilated variants and a parallel implementation.
Key results
- On CIFAR-10, the AOC accurate configuration reaches 91.5% clean accuracy with 2.3M parameters (0% provable accuracy at eps=36/255).
- Scaling AOC to 41.3M parameters yields 64.3% provable accuracy on CIFAR-10 at eps=36/255, competitive with SLL and AOL.
- On ResNet-34 with 224x224 inputs at batch 128, AOC trains at 1.75x Conv2D time versus 2.85x for BCOP, 4.86x for SOC, and 4.27x for Cayley; at batch 512 the overhead falls to 1.13x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AOC accurate rather than AOC robust training | AOC · CIFAR-10 | accuracy: 74.0 -> 91.5 | Helped |
| Use AOC robust rather than AOC accurate training | AOC · CIFAR-10 | provable accuracy (ε = 36/255): 00.0 -> 64.3 | Helped |
| Use the AOC robust* recipe with extra data | AOC · CIFAR-10 | accuracy: 74.0 -> 85.3 | Helped |
| Use the AOC robust* recipe with extra data | AOC · CIFAR-10 | provable accuracy (ε = 36/255): 64.3 -> 75.0 | Helped |
| Use AOC accurate rather than AOC robust training | AOC · ImageNet-1K | accuracy: 42.1 -> 68.2 | Helped |
| Use AOC robust rather than AOC accurate training | AOC · ImageNet-1K | provable accuracy (ε = 36/255): 00.0 -> 26.3 | Helped |
| Increase the AOC batch size | AOC on ResNet-34 · ResNet-34 training, batch size 128 to 512 | training-time overhead: 1.75x -> 1.13x | Helped |
| Use Cholesky instead of Björck and Bowie orthogonalization (ablation) | AOC · Network from the cited training setup | accuracy | Hurt |
| Use the exponential orthogonalization method (ablation) | AOC · Network from the cited training setup | training speed | Hurt |
| Use QR instead of Björck and Bowie orthogonalization (ablation) | AOC · Network from the cited training setup | accuracy | No clear effect |
| Reduce Björck and Bowie iterations from 25 to 12 (ablation) | AOC · Orthogonality unit tests | orthogonality: 25 iterations -> 12 | No clear effect |
Limitations
- Does not improve expressiveness of the underlying BCOP building block; provable accuracy is low for the accuracy-oriented configuration.
- Some configurations (e.g., kernel size smaller than stride) admit no orthogonal convolution; BCOP cannot build 1-channel grouped kernels (c>=2 required).
Experimental setup
Datasets: CIFAR-10 ImageNet-1K
Benchmarks: CIFAR-10 ImageNet-1K
Metrics: clean accuracy provable accuracy training time training memory test time
Baselines: BCOP SOC Cayley SLL AOL LOT GloRo Local-Lip-B CPL Li-Resnets
Builds on: BCOP RKO SC-Fac SOC
Closest papers by meaning
- Scaling-up Diverse Orthogonal Convolutional Networks by a Paraunitary Framework (ICML 2022 · similarity 0.91)
- Orthogonal Convolutional Neural Networks (CVPR 2020 · similarity 0.90)
- Constructing Orthogonal Convolutions in an Explicit Manner (ICLR 2022 · similarity 0.90)
- Skew Orthogonal Convolutions (ICML 2021 · similarity 0.89)
- Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks (NeurIPS 2025 · similarity 0.85)
- KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.