Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Inverse Transformation Search (ITS), a model-agnostic inference-time search that undoes affine transformations of an input by traversing a sparsified inverse transformation tree scored with a Monte-Carlo-dropout group-induced confidence, giving zero-shot pseudo-invariance without augmentation.
Problem
Deep classifiers are not robust to spatially transformed (rotated, scaled, sheared) inputs; data augmentation and hard-coded equivariant architectures are limited in coverage and engineering cost.
Approach
At inference, search over subgroup transformations of the input, scoring each candidate by a group-induced confidence (negative curvature of smoothed MC-dropout energy along the orbit), keeping k parallel hypotheses with one per label, to find the canonical form using only the classifier's logits.
Key results
- On SI-Rotation with ViT-B16, ITS raises acc@1 from 38.5 to 49.0 and acc@5 to 70.9 (best-of-hypotheses variant), beating rotation+scale augmentation (41.0) and augmented fine-tuning with segmentation masks (46.2).
- On affine-transformed MNIST, Fashion-MNIST and GTSRB, ITS^3 reaches 88.89% on MNIST versus 50.15% for a plain CNN and 56.59% for ETN.
- ITS gives no gain on SI-Size and SI-Location, and reduces accuracy on SI-Location for EfficientNet (15.7 to 11.4 acc@1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ITS at inference | CNN classifier · MNIST affine canonicalisation | accuracy: 50.15$\pm$9.29 -> 88.89$\pm$13.86 | Helped |
| use 5-layer rather than 3-layer ITS (ablation) | CNN classifier · MNIST affine canonicalisation | accuracy: 88.89$\pm$13.86 -> 89.81$\pm$13.95 | No clear effect |
| apply ITS at inference | CNN classifier · F-MNIST affine canonicalisation | accuracy: 25.91$\pm$5.93 -> 38.47$\pm$7.77 | Helped |
| use 5-layer rather than 3-layer ITS (ablation) | CNN classifier · F-MNIST affine canonicalisation | accuracy: 38.47$\pm$7.77 -> 37.58$\pm$7.56 | No clear effect |
| apply ITS at inference | CNN classifier · GTSRB affine canonicalisation | accuracy: 32.01$\pm$6.59 -> 67.09$\pm$11.53 | Helped |
| use 5-layer rather than 3-layer ITS (ablation) | CNN classifier · GTSRB affine canonicalisation | accuracy: 66.44$\pm$11.48 -> 67.09$\pm$11.53 | No clear effect |
| apply ITS at inference | ViT-B16 · SI-Rotation | acc@1: +10.5 | Helped |
| apply ITS at inference | ViT-B16 · SI-Size | acc@1: +0.2 | No clear effect |
| apply ITS at inference | ViT-B16 · SI-Location | acc@1: -3.6 | Hurt |
| apply ITS at inference | EfficientNet-B0 · SI-Rotation | acc@1: +9.3 | Helped |
| apply ITS at inference | ConvNeXt-tiny · SI-Rotation | acc@1: +12.3 | Helped |
| apply ITS at inference | EfficientNet-B0 · SI-Size | acc@5: -6.8 | Hurt |
Surprising findings
- On SI-Size and SI-Location ITS fails to improve and sometimes degrades the ViT baseline, attributed to the search focusing on wrong objects when translation is used.
- Frobenius distance and accuracy do not always agree; ITS^5 has lowest transformation error but ITS^3 has higher accuracy on some datasets.
Limitations
- Heavily perturbed inputs may admit multiple plausible hypotheses, and false hypotheses can be reinforced down the search tree.
- Assumes the transformation decomposes into mostly Abelian subgroup sequences; the Abelian assumption is only partially relaxed.
- Substantial inference runtime cost, making real-time applications difficult.
Experimental setup
Datasets: MNIST Fashion-MNIST GTSRB ImageNet SI-Score
Benchmarks: MNIST rotated test sets Fashion-MNIST affine test sets GTSRB affine test sets SI-Score (SI-Rotation, SI-Size, SI-Location)
Metrics: accuracy acc@1 acc@5 Frobenius distance
Baselines: CNN STN STN^3 DTN ETN GCNN RotDCF ViT rotation+scale augmentation EfficientNet ConvNeXt
Builds on: spatial transformer networks group equivariant CNNs energy-based models Monte-Carlo dropout
Closest papers by meaning
- Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers (ICML 2021 · similarity 0.88)
- Revisiting Spatial Invariance with Low-Rank Local Connectivity (ICML 2020 · similarity 0.88)
- Grounding inductive biases in natural images: invariance stems from variations in data (NeurIPS 2021 · similarity 0.87)
- From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication (ICLR 2024 · similarity 0.85)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.84)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.