Research map / Large language models / Language, safety and interpretability
GLM-130B: An Open Bilingual Pre-trained Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| INT4 weight quantization without post-training (activations kept FP16) (ablation) | GLM-130B · LAMBADA zero-shot | accuracy: 80.21 -> 79.47 (-0.74) | No clear effect |
| INT4 weight quantization without post-training (activations kept FP16) (ablation) | GLM-130B · MMLU 5-shot | accuracy: 44.75 -> 44.80 (+0.05) | No clear effect |
| INT8 weight quantization (activations FP16) (ablation) | GLM-130B · LAMBADA zero-shot | accuracy: 80.21 -> 80.21 | No clear effect |
| Embedding layer gradient shrink (alpha=0.1) (ablation) | GLM-130B training · Training stability (loss spikes) | loss spikes / divergence | Helped |
| DeepNorm Post-LN initialization instead of Pre-LN/Sandwich-LN (ablation) | GLM-130B training · Training stability | training collapse | Helped |
| RoPE instead of ALiBi for positional encoding (base model) (ablation) | GLM-Base 110M · Test PPL | test perplexity: 24.14 (ALiBi) -> 22.95 (RoPE) | Helped |
| GeGLU FFN added on top of RoPE (base model) (ablation) | GLM-Base 110M · Test PPL | test perplexity: 22.95 -> 22.31 | Helped |
| Embedding-layer gradient shrink with norm-based FP16 vs embedding norm (BLOOM-style) and PB-Relax | GLM-130B training · Training stability | loss spikes | Mixed |
| GLM-130B vs GPT-3 175B on LAMBADA zero-shot | GLM-130B · LAMBADA zero-shot | accuracy: +4.0 points (80.2 vs 76.2) | Helped |
| Bidirectional GLM backbone vs unidirectional attention variant (ablation) | GLM-130B · LAMBADA zero-shot | accuracy: 80.2 bidirectional vs 75.3 unidirectional | Helped |
| GLM-130B vs OPT-175B and GPT-3 on CrowS-Pairs | GLM-130B · CrowS-Pairs overall | bias score (lower better): 65.8 vs 67.2 (GPT-3) and 69.5 (OPT-175B) | Helped |
| GLM-130B vs GPT-3 on ETHOS multiclass few-shot | GLM-130B · ETHOS hate speech detection | F1: 85.8 vs 67.2 (GPT-3), 81.2 (OPT-175B) | Helped |
Closest papers by meaning
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.87)
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.86)
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.86)
- GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models (ICML 2025 · similarity 0.86)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.86)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.