Research map / Architectures and efficiency / Pruning and efficient tuning
OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OrthCaps, a capsule network that combines a similarity-based pruned capsule layer, non-iterative orthogonal sparse attention routing (alpha-entmax), and Householder-parameterized orthogonal weight matrices to cut redundancy and parameters.
Problem
Capsule networks suffer from redundant capsules, high parameter counts and computational cost, and dynamic routing both requires iterations and reintroduces redundancy in deeper layers via fully connected, non-orthogonal transformations.
Approach
Replace dynamic routing with non-iterative sparse attention routing (alpha-entmax), prune redundant capsules by cosine similarity after ordering by L2 norm, and enforce orthogonality of routing weight matrices via Householder decomposition without penalty terms.
Key results
- OrthCaps-S reaches 96.26% on SVHN and 86.84% on CIFAR10 with 105.5K parameters, versus 8388K for CapsNet.
- OrthCaps-D achieves 90.56% on CIFAR10 and 70.56% on CIFAR100 with 574K parameters, about 23x fewer than DeepCaps.
- Ablation: pruning raises OrthCaps-S CIFAR10 accuracy from 85.32% to 86.84% while reducing parameters from 157K to 105K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OrthCaps-S instead of Efficient-Caps | OrthCaps-S · MNIST | accuracy: 99.58 -> 99.68 | Helped |
| Use OrthCaps-S instead of DA-CapsNet | OrthCaps-S · SVHN | accuracy: 94.82* -> 96.26 | Helped |
| Use OrthCaps-S instead of DA-CapsNet | OrthCaps-S · smallNORB | accuracy: 98.26* -> 98.30 | No clear effect |
| Use OrthCaps-D instead of ResNet-18 | OrthCaps-D · CIFAR10 | accuracy: 95.10 -> 90.56 | Hurt |
| Use OrthCaps-D instead of ResNet-18 | OrthCaps-D · CIFAR100 | accuracy: 77.60 -> 70.56 | Hurt |
| Use attention routing rather than dynamic routing with α-entmax and orthogonality (ablation) | OrthCaps-S · CIFAR10 | accuracy: 70.01 -> 86.84 | Helped |
| Replace α-entmax with softmax in attention routing (ablation) | OrthCaps-S · CIFAR10 | accuracy: 86.84 -> 83.44 | Hurt |
| Replace α-entmax with softmax in attention routing (ablation) | OrthCaps-S · MNIST | FPS: 1639 -> 1785 | Helped |
| Add the pruning layer (ablation) | OrthCaps-S · MNIST | accuracy: 99.53 -> 99.68 | Helped |
| Add the pruning layer (ablation) | OrthCaps-S · CIFAR10 | accuracy: 85.32 -> 86.84 | Helped |
| Vary the similarity threshold θ (ablation) | OrthCaps-S · MNIST, SVHN, and CIFAR10 | accuracy: 38.25; 43.60; 90.09; 97.46; 99.68; 99.63; 99.61; 99.53 | Mixed |
| Orthogonalize WQ, WK, and WV rather than none (ablation) | OrthCaps-S · CIFAR10 | accuracy: 84.25 -> 86.84 | Helped |
Surprising findings
- Pruning improved accuracy while reducing parameters, both for OrthCaps-S and for CapsNet.
- Attention routing with orthogonality was reported to be faster than dynamic routing despite the added orthogonal and entmax computation.
Limitations
- Orthogonality is not enforced on the full coupling matrix C, only on the weight matrices, so routing outputs are not guaranteed orthogonal.
- Several baselines are results taken from original papers without official code, so comparisons are not fully controlled.
- OrthCaps-D has higher FLOPS than several baselines (3345M vs 2687M for DeepCaps).
Experimental setup
Datasets: MNIST SVHN smallNORB CIFAR10 CIFAR100 Fashion-MNIST
Benchmarks: MNIST SVHN smallNORB CIFAR10 CIFAR100 Fashion-MNIST
Metrics: accuracy parameter count FLOPS FPS attack time query count orthogonality metric
Baselines: Efficient-Caps CapsNet Matrix-CapsNet with EM routing AR CapsNet DA-CapsNet AA-CapsNet CapProNet DeepCaps ResNet-18 VGG-16 OCNN
Builds on: CapsNet (dynamic routing) Matrix-CapsNet with EM routing Efficient-CapsNet Householder orthogonal decomposition alpha-entmax
Closest papers by meaning
- STAR-Caps: Capsule Networks with Straight-Through Attentive Routing (NeurIPS 2019 · similarity 0.89)
- Ladder Capsule Network (ICML 2019 · similarity 0.89)
- DeepCaps: Going Deeper With Capsule Networks (CVPR 2019 · similarity 0.88)
- Self-Routing Capsule Networks (NeurIPS 2019 · similarity 0.87)
- Introducing Routing Uncertainty in Capsule Networks (NeurIPS 2020 · similarity 0.87)
- ParseCaps: An Interpretable Parsing Capsule Network for Medical Image Diagnosis (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.