Research map / Vision-language and multimodal / CLIP and zero-shot
The Impact of Geometric Complexity on Neural Collapse in Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that embedding geometric complexity (GC) controls neural collapse (NC) during pre-training and that lower source-GC yields more target-class NC and better few-shot transfer.
Problem
Explaining why pre-trained representations transfer well, by linking loss-surface flatness, geometric complexity, and neural collapse, especially for few-shot transfer to unseen classes.
Approach
Uses the Dirichlet-energy-style geometric complexity (expected squared Frobenius norm of the embedding Jacobian) as a bridge: via a Poincaré inequality, GC bounds the class-distance normalized variance (NC), yielding a "geometric collapse" proxy, a GC-based generalization bound, and a transfer bound linking source GC to target NC.
Key results
- Lower embedding GC coincides with lower NC and higher test accuracy across learning rate, batch size and L2 sweeps for VGG-13 on CIFAR-10.
- Empirical GC estimates remain stable under subsampling of batch examples, Jacobian entries, and output dimensions on ResNet-18 trained on CIFAR-100.
- Lower source GC on CIFAR-FS with ResNet-18 yields lower target-class NC and higher 5-shot transfer accuracy across sweeps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase learning rate during pre-training | VGG-13 · CIFAR-10 | neural collapse (NC) | Helped |
| decrease batch size during pre-training | VGG-13 · CIFAR-10 | neural collapse (NC) | Helped |
| increase L2 regularization during pre-training | VGG-13 · CIFAR-10 | neural collapse (NC) | Helped |
| increase learning rate during pre-training | ResNet-18 · CIFAR-FS 5-shot transfer | transfer accuracy | Helped |
| decrease batch size during pre-training | ResNet-18 · CIFAR-FS 5-shot transfer | transfer accuracy | Helped |
| increase L2 regularization during pre-training | ResNet-18 · CIFAR-FS 5-shot transfer | transfer accuracy | Helped |
| increase learning rate during pre-training | VGG-16 · mini-ImageNet 5-shot transfer | transfer accuracy | Helped |
| increase explicit geometric-complexity regularization | VGG-13 · CIFAR-10 | neural collapse (NC) | Helped |
| increase learning rate during training | ResNet-18 · CIFAR-10 | neural collapse (NC) | Helped |
| decrease batch size during training | ResNet-18 · CIFAR-10 | neural collapse (NC) | Helped |
| increase L2 regularization during training | ResNet-18 · CIFAR-10 | neural collapse (NC) | Helped |
| increase learning rate during training | VGG-11 · MNIST | neural collapse (NC) | Helped |
Surprising findings
- Explicit GC regularization had to be applied at the logit layer rather than the embedding layer because embedding-GC gradients were prohibitively expensive, and only one seed ran.
Limitations
- Extension to language models/LLMs is unclear because token-prediction classification makes neural collapse implausible given vocabulary size exceeding embedding dimension.
- Transfer bound requires a granularity compatibility condition between source and target classes that is hard to verify in practice.
- Poincaré constant c must be estimated (c≈1000 reported); embedding-GC gradient regularization is computationally prohibitive.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST CIFAR-FS mini-ImageNet
Benchmarks: CIFAR-10 CIFAR-100 MNIST Fashion-MNIST CIFAR-FS mini-ImageNet
Metrics: neural collapse (CDNV) geometric complexity geometric collapse 5-shot ridge regression transfer accuracy test accuracy
Baselines: nearest-mean classifier
Builds on: neural collapse (Papyan et al.) geometric complexity (Lotfi/Ramasinghe-style logit GC) CDNV neural collapse measure Poincaré inequality
Closest papers by meaning
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.87)
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.87)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.86)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.