Research map / Architectures and efficiency / RNNs and normalization
Improving Equivariant Model Training via Constraint Relaxation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes relaxing the equivariance constraint of intermediate layers during training via an additive unconstrained linear term whose weight is cyclically scheduled and regularized by a Lie derivative penalty, then projecting back to an exactly equivariant model at inference.
Problem
Equivariant neural networks are hard to optimize and need careful hyperparameter tuning, even when the symmetry is correctly specified and known exactly.
Approach
Add an unconstrained linear residual term to each equivariant layer, controlled by a scalar theta that is cyclically scheduled to zero by the end of training, with Lie derivative and activation-norm regularizers keeping the model near the equivariant space; at test time theta=0 yields an exactly equivariant model with the baseline architecture.
Key results
- On ModelNet40, the relaxed training framework improves VN-PointNet and VN-DGCNN accuracy over standard training, with a larger gain for the smaller PointNet.
- On MD17, Equiformer+Ours lowers energy MAE on aspirin (5.2 vs 5.3 meV), ethanol (2.0 vs 2.2 meV) and salicylic acid (4.1 vs 4.5 meV), and force MAE on aspirin (7.1 vs 7.2) and ethanol (2.9 vs 3.1).
- On the smoke plume RMSE, RSteer+Ours achieves the best results in all four settings (e.g., rotation Domain 0.58 vs 0.67 for RSteer).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| relax equivariance during training and project back at inference | Equiformer · MD17 (Aspirin, Benzene, Ethanol, Salicylic acid) | energy and force MAE: Aspirin energy 5.3 -> 5.2; forces 7.2 -> 7.1; Benzene energy 2.2 -> 2.2; forces 6.6 -> 6.6; Ethanol energy 2.2 -> 2.0; f | Helped |
| progressively constrain approximate equivariance during training | RSteer · Smoke flow rotation Future and Domain; scale Future and Domain | RMSE: Rotation Future 0.80 -> 0.79; Rotation Domain 0.67 -> 0.58; Scale Future 0.70 -> 0.62; Scale Domain 0.73 -> 0.67 | Helped |
| add Lie derivative regularization (ablation) | relaxed VN-PointNet and VN-DGCNN · ModelNet40 classification | test accuracy | Helped |
| add regularization terms to constrain relaxed activations and equivariance error (ablation) | relaxed VN-PointNet and VN-DGCNN · ModelNet40 classification | test accuracy | Helped |
| schedule θ instead of fixing it to a constant low value (ablation) | relaxed VN-PointNet and VN-DGCNN · ModelNet40 classification | test accuracy | Helped |
| use constant θ instead of the proposed θ schedule (ablation) | relaxed VN-PointNet · ModelNet40 classification (300 points) | test accuracy after equivariant projection | Hurt |
| remove regularization (ablation) | relaxed VN-PointNet and VN-DGCNN · ModelNet40 classification | test accuracy after projection | Hurt |
Surprising findings
- Without the Lie derivative regularizer, test accuracy drops, and a constant theta without scheduling shows a sudden performance drop after projection to the equivariant space.
Limitations
- Assumes the equivariant network imposes the constraint in every intermediate layer; does not cover all equivariant architectures.
- Regularizers apply only to matrix Lie groups or discrete finite groups.
- No theoretical analysis of optimization error is provided.
Experimental setup
Datasets: ModelNet40 N-body particle simulation MD17 PhiFlow smoke plume simulation
Benchmarks: ModelNet40 N-body particle simulation MD17 synthetic smoke plume dataset
Metrics: accuracy MAE RMSE
Baselines: VN-PointNet VN-DGCNN SEGNN Equiformer MLP ConvNet Equiv RPP LIFT RSteer Equiv-Adapt Equi-Tuning
Builds on: Residual Pathway Priors Vector Neurons SEGNN Equiformer Approximately equivariant steerable CNNs
Closest papers by meaning
- Learning (Approximately) Equivariant Networks via Constrained Optimization (NeurIPS 2025 · similarity 0.93)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.92)
- Approximately Equivariant Neural Processes (NeurIPS 2024 · similarity 0.88)
- On Universality Classes of Equivariant Networks (NeurIPS 2025 · similarity 0.88)
- On the hardness of learning under symmetries (ICLR 2024 · similarity 0.88)
- A Characterization Theorem for Equivariant Networks with Point-wise Activations (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.