Research map / Large language models / Language, safety and interpretability
Can Unconditional Language Models Recover Arbitrary Sentences?
Nishant Subramani, Samuel Bowman, Kyunghyun Cho
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase reparametrized-space dimension from 512 to 4096 (ablation) | Large LSTM trained on 50M Gigaword sentences · Gigaword held-out sentences | BLEU: 54.8 -> 99.8 | Improved |
| Increase reparametrized-space dimension from 512 to 2048 (ablation) | Medium LSTM trained on 50M Gigaword sentences · Gigaword held-out sentences | BLEU: 41.1 -> 91.2 | Improved |
| Increase reparametrized-space dimension from 512 to 8192 (ablation) | Small LSTM trained on 50M Gigaword sentences · Gigaword held-out sentences | BLEU: 40.0 -> 81.1 | Improved |
| Increase language-model training data from 10M to 50M sentences (ablation) | Fixed LSTM language models with reparametrized sentence conditioning · Gigaword held-out sentences | recoverability | Improved |
| Use Adam instead of nonlinear conjugate gradient for forward estimation (ablation) | Reparametrized sentence-space optimization · Gigaword held-out sentences | BLEU: do not exceed 1.0 BLEU | Hurt |
| Increase beam width from 5 to 10 (ablation) | Large LSTM, |Z|=4096 · Gigaword held-out sentences | BLEU: 99.8 -> 99.8 | No clear effect |
| Increase beam width from 5 to 20 (ablation) | Large LSTM, |Z|=4096 · Gigaword held-out sentences | BLEU: 99.8 -> 99.5 | No clear effect |
| Use |Z| greater than the model dimension (ablation) | Well-trained medium and large LSTMs · Gigaword held-out sentences | recoverability | No clear effect |
| Increase sentence length | Reparametrized sentence-space recovery · Gigaword held-out sentences | recoverability | Hurt |
| Condition the medium and large models on optimized sentence vectors with |Z| at least the model dimension | Fixed pretrained LSTM language models · IWSLT16 English-to-German validation sentences | BLEU | Improved |
| Recover randomly sampled token sequences longer than 25 subword units | Large LSTM with reparametrized sentence conditioning · Random sequences | BLEU | Hurt |
Closest papers by meaning
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (ICLR 2017 · similarity 0.86)
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (ICLR 2018 · similarity 0.86)
- No Training Required: Exploring Random Encoders for Sentence Classification (ICLR 2019 · similarity 0.86)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.85)
- An efficient framework for learning sentence representations (ICLR 2018 · similarity 0.85)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
