Research map / Architectures and efficiency / RNNs and normalization
Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU instead of linear activation | Two-layer contrastive learning model · Synthetic token-sequence dataset, P > 1 | matching score | Helped |
| Use linear activation instead of ReLU | Two-layer contrastive learning model · Synthetic token-sequence dataset, P=1 | matching score | Helped |
| Increase over-parameterization | ReLU contrastive learning model · Synthetic token-sequence dataset, P > 1 | matching score | Helped |
| Increase over-parameterization | Linear contrastive learning model · Synthetic token-sequence dataset | matching score | No clear effect |
| Use InfoNCE instead of quadratic contrastive loss | Two-layer contrastive learning model · Synthetic token-sequence dataset | matching score | Helped |
| Use nonlinear activation instead of linear activation | Two-layer network with multiple filters per receptive field · Contrastive learning dynamics | diverse feature learning | Helped |
| Add top-level support for features discriminative of the global variable | Two-layer contrastive learning model · Synthetic token-sequence dataset | learning of generator-related tokens | Helped |
Closest papers by meaning
- Self-supervised contrastive learning performs non-linear system identification (ICLR 2025 · similarity 0.88)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.87)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.86)
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables (NeurIPS 2025 · similarity 0.86)
- When Can We Approximate Wide Contrastive Models with Neural Tangent Kernels and Principal Component Analysis? (AAAI 2025 · similarity 0.85)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.