Research map / Label-efficient and robust learning / Self-supervised and contrastive
On Masked Pre-training and the Marginal Likelihood
Pablo Moreno-Muñoz, Pol G. Recasens, Søren Hauberg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of random masks from P=1 to P=100 (ablation) | Cumulative MPT loss in PPCA · PPCA | approximation to true LML: -60.44 ± 0.47 -> -60.34 ± 0.03; true LML (-60.34) | Improved |
| Increase the number of random masks from P=1 to P=10 (ablation) | Cumulative MPT loss in PPCA · PPCA | approximation to true LML: -60.44 ± 0.47 -> -60.22 ± 0.12; true LML (-60.34) | Improved |
| Fix the mask size instead of varying masking rates (ablation) | MPT training · PPCA | bias of the LML estimate | Hurt |
| Use fixed mask size during training (ablation) | MPT training · PPCA | LML maximization | No clear effect |
| Train with approximate self-predictive conditional probabilities (ablation) | MPT · MNIST and Fashion-MNIST subsets | LML | No clear effect |
| Use MPT instead of variational inference with ELBO (ablation) | Linear VAE · MNIST and Fashion-MNIST subsets | LML | No clear effect |
| Pre-train BERT rather than use random initialization | BERT · GLUE AX | area under MPT curve: -5245.31 -> -1715.75 | Improved |
| Pre-train BERT rather than use random initialization | BERT · GLUE CoLA | area under MPT curve: -5283.52 -> -1657.68 | Improved |
| Pre-train BERT rather than use random initialization | BERT · GLUE QNLI | area under MPT curve: -5343.98 -> -1770.28 | Improved |
| Pre-train BERT rather than use random initialization | BERT · GLUE MRPC | area under MPT curve: -5362.21 -> -1773.45 | Improved |
Closest papers by meaning
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.89)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.88)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.87)
- MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Masking meets Supervision: A Strong Learning Alliance (CVPR 2025 · similarity 0.87)
- Instruction-Following Pruning for Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
