Research map / Architectures and efficiency / RNNs and normalization
Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the original encoder with clean-path residuals and set norm | Deep Sets · MNIST Var | MSE: 5.5545 ± 0.0014 -> 0.3441 ± 0.0036 | Helped |
| replace the original encoder with clean-path residuals and set norm | Set Transformer · Point Cloud | accuracy: 0.7134 ± 0.0030 -> 0.8258 ± 0.0046 | Helped |
| increase depth from 3 to 50 layers (ablation) | Deep Sets · Point Cloud | accuracy: 0.7755 ± 0.0051 -> 0.0409 ± 0.0000 | Hurt |
| increase depth from 2 to 16 layers (ablation) | Set Transformer · MNIST Var | MSE: 0.6151 ± 0.0072 -> 6.2663 ± 0.0036 | Hurt |
| add set norm instead of no normalization (ablation) | 50-layer Deep Sets · Point Cloud | CE: 3.609 ± 0.000 -> 1.542 ± 0.086 | Helped |
| add set norm instead of no normalization (ablation) | 50-layer Deep Sets · MNIST Var | MSE: 5.555 ± 0.001 -> 0.259 ± 0.003 | Helped |
| add layer norm instead of no normalization (ablation) | Set Transformer · Normal Var | MSE: 0.0023 ± 0.0006 -> 0.0801 ± 0.0076 | Hurt |
| add layer norm instead of no normalization (ablation) | Set Transformer · Hematocrit | MSE: 18.7436 ± 0.0148 -> 19.0904 ± 0.1003 | Hurt |
| use clean-path rather than non-clean-path residuals (ablation) | Deep Sets with set norm · Normal Var | MSE: 0.2094 ± 0.1115 -> 0.0198 ± 0.0041 | Helped |
| use clean-path rather than non-clean-path residuals (ablation) | Set Transformer with set norm · Normal Var | MSE: 0.0249 ± 0.0112 -> 0.0030 ± 0.0000 | Helped |
| replace equivariant residual connections with mean-aggregated residual connections (ablation) | Deep Sets · Normal Var | MSE: 0.0198 ± 0.0041 -> 0.8811 ± 0.1824 | Hurt |
| replace equivariant residual connections with max-aggregated residual connections (ablation) | Deep Sets · Point Cloud | CE: 0.7096 ± 0.0049 -> 0.8758 ± 0.0196 | Hurt |
Closest papers by meaning
- On Universal Equivariant Set Networks (ICLR 2020 · similarity 0.88)
- Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (ICLR 2022 · similarity 0.87)
- Permutation Equivariance of Transformers and Its Applications (CVPR 2024 · similarity 0.87)
- REVISITING MULTI-PERMUTATION EQUIVARIANCE THROUGH THE LENS OF IRREDUCIBLE REPRESENTATIONS (ICLR 2025 · similarity 0.85)
- Polynomial Width is Sufficient for Set Representation with High-dimensional Features (ICLR 2024 · similarity 0.85)
- Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.