Research map / Probabilistic and scientific modeling / Variational and generative sampling
MLE-Guided Parameter Search for Task Loss Minimization in Neural Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MGS to minimize LM task loss | GPT-2 117M · Wikitext-103 test, greedy decoding | Nonterm: .344 (.063) -> .012 (.003) | Helped |
| Apply MGS to minimize LM task loss | GPT-2 117M · Wikitext-103 test, greedy decoding | Repetition: .530 (.062) -> .046 (.009) | Helped |
| Apply MGS to minimize LM task loss | GPT-2 117M · Wikitext-103 test, greedy decoding | Perplexity: 21.3 (0.2) -> 22.0 (0.1) | Hurt |
| Use MGS rather than MLE for SBleu fine-tuning | Transformer IWSLT De-En · IWSLT '14 De→En test | BLEU: 34.71 -> 35.03 | Helped |
| Fine-tune with MGS-Edit | Transformer IWSLT De-En · IWSLT '14 De→En test | Edit: 50.74 -> 49.45 | Helped |
| Fine-tune with MGS-Edit | Transformer IWSLT De-En · IWSLT '14 De→En test | BLEU: 34.71 -> 34.73 | No clear effect |
| Replace the mixture proposal with only its zero-mean component (ablation) | MGS on GPT-2 117M · Wikitext-103 validation | Task-loss: 59.2 -> 143.0 | Hurt |
| Replace the mixture proposal with only its MLE-mean component (ablation) | MGS on GPT-2 117M · Wikitext-103 validation | Task-loss: 59.2 -> 141.6 | Hurt |
| Replace the mixture proposal with only its zero-mean component (ablation) | MGS on GPT-2 117M · Wikitext-103 validation | Nonterm: .013 -> .348 | Hurt |
| Replace the mixture proposal with only its MLE-mean component (ablation) | MGS on GPT-2 117M · Wikitext-103 validation | Nonterm: .013 -> .351 | Hurt |
| Make candidate weights too uniform or too peaked by changing alpha (ablation) | MGS · Text completion | Task-loss | Hurt |
Closest papers by meaning
- Sentence-Wise Smooth Regularization for Sequence to Sequence Learning (AAAI 2019 · similarity 0.89)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.87)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.86)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.86)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.86)
- SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.