Research map / Architectures and efficiency / RNNs and normalization
Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari S. Morcos, Ali Farhadi, Ludwig Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SwitchBack int8, keeping weight-gradient computation in higher precision | CLIP ViT-Huge · ImageNet zero-shot | accuracy vs bfloat16: 0.1 percentage points | No clear effect |
| Use LLM.int8() for training | CLIP ViT-Huge · ImageNet zero-shot | accuracy vs bfloat16: 5.9 percentage points | Hurt |
| Replace transformer linear layers with SwitchBack int8 | CLIP ViT-Huge · CLIP training | speedup: 13%; 25% | Improved |
| Use SwitchBack simulated float8 | CLIP ViT-Huge · ImageNet zero-shot | accuracy vs bfloat16: 0.1 percentage points | No clear effect |
| Use tensor-wise float8 quantization for inputs, weights, and gradients without other modifications | CLIP ViT-Large · CLIP training | training stability | Hurt |
| Add zero-initialized layer-scale to tensor-wise float8 training (ablation) | CLIP ViT-Large · CLIP training | training stability | Improved |
| Add gradient clipping to tensor-wise float8 training (ablation) | CLIP ViT-Large · CLIP training | training stability | No clear effect |
| Add KQ layernorm to tensor-wise float8 training (ablation) | CLIP ViT-Large · CLIP training | training stability | No clear effect |
| Replace tensor-wise weight quantization with row-wise and column-wise weight quantization (ablation) | SwitchBack · CLIP training at tested scales | accuracy | No clear effect |
| Save quantized rather than full-precision tensors for the backward pass (ablation) | SwitchBackM · CLIP ViT-Huge | accuracy | No clear effect |
| Add AdaFactor-style update clipping to AdamW | CLIP ViT-Huge · CLIP training | loss spikes | Improved |
| Use update clipping instead of gradient clipping | AdamW on CLIP ViT-Huge · ImageNet zero-shot | accuracy | Improved |
Closest papers by meaning
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.88)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.87)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.87)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
