Research map / Label-efficient and robust learning / Recognition and long-tail
Global Minimizers of Sigmoid Contrastive Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes global minimizers of the SigLIP sigmoid loss with trainable inverse temperature and bias as (m, relative bias)-constellations, linking them to spherical codes, retrieval, the modality gap, and a relative-bias parameterization.
Problem
Understanding which representations minimize the sigmoid contrastive loss with trainable inverse temperature and bias in the practical regime d << N << 2^d, including retrieval success, modality gap, and required embedding dimension.
Approach
Shows that zero-loss (global minimum) configurations of the sigmoid loss with trainable inverse temperature and bias are exactly (m, relative bias)-constellations, i.e., inner-product separation between positive and negative pairs; connects these to spherical codes, derives retrieval and modality-gap guarantees, and proposes a relative-bias reparameterization of the loss.
Key results
- Any (m, relative bias)-constellation is a zero-loss global minimizer of the sigmoid loss with trainable temperature and bias, and nearest-neighbor search on it gives perfect retrieval.
- All 8 tested SigLIP Hugging Face models show perfect linear separability of image and text embeddings on ImageNet, consistent with the modality gap theorem.
- Relative-bias parameterization of sigmoid loss converges faster than the standard bias parameterization on synthetic data, and trainable temperature and bias outperform fixed ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use explicit relative-bias parameterization instead of the usual bias parameterization | Sigmoid loss training with Adam on synthetic data · Synthetic data | loss convergence speed | Helped |
| Train inverse temperature and bias rather than fixing both at low inverse temperature (ablation) | Sigmoid-loss synchronization of multiple embeddings and with a locked encoder · Synthetic experiments | inner-product separation and margin | Helped |
| Train inverse temperature and bias rather than fixing both at high inverse temperature (ablation) | Sigmoid-loss synchronization · Synthetic experiments | inner-product separation | Helped |
| Use the usual bias parameterization instead of explicit relative bias (ablation) | Sigmoid-loss synchronization · Synthetic experiments | loss convergence speed | Hurt |
| Use the usual bias parameterization instead of explicit relative bias (ablation) | Sigmoid-loss synchronization · Synthetic experiments | margin | Hurt |
| Increase embedding dimension | SigLIP models · SigLIP models | margin: +0.948 Pearson correlation; .926 Spearman coefficient | Helped |
Surprising findings
- In practice Adam on the standard sigmoid loss drives the relative bias toward zero, limiting the set of learned configurations.
- Any zero-loss configuration with positive-pair and negative-pair separation also minimizes the triplet loss.
Limitations
- No rigorous convergence guarantees for first-order methods such as Adam.
- The combinatorial constellation-size problem is not fully resolved.
- Relative-bias parameterization is only tested on synthetic data, not real data.
Experimental setup
Datasets: ImageNet synthetic data
Benchmarks: ImageNet
Metrics: margin relative bias linear separability misclassification rate loss convergence nearest neighbor retrieval
Baselines: InfoNCE fixed temperature and bias sigmoid loss bias parameterization sigmoid loss triplet loss
Builds on: SigLIP CLIP InfoNCE Double-Constant Embedding Model
Closest papers by meaning
- A Global Geometric Analysis of Maximal Coding Rate Reduction (ICML 2024 · similarity 0.84)
- The Complexity of Finding Local Optima in Contrastive Learning (NeurIPS 2025 · similarity 0.83)
- The Star Geometry of Critic-Based Regularizer Learning (NeurIPS 2024 · similarity 0.83)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.83)
- Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.82)
- Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.