Research map / Probabilistic and scientific modeling / Molecules and proteins
Predictive Querying for Autoregressive Neural Sequence Models
Alex James Boyd, Sam Showalter, Stephan Mandt, Padhraic Smyth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use importance sampling instead of beam search | GPT-2 predictive hitting-time queries · WikiText, K=4, budget O(10^3) | median RAE: 82\% -> 13\% | Improved |
| use hybrid beam-plus-sampling estimator instead of pure importance sampling | LSTM sequence models · Reviews, Mobile Apps, MOOCs, Shakespeare, K=3..11 | median RAE | Improved |
| use importance sampling or hybrid instead of beam search as K grows | LSTM sequence models · 3 of 4 datasets, K=3..11 | median RAE | Improved |
| raise model temperature T (higher entropy) | Beam search · Mobile Apps, tau(.)=4 | median RAE: error converges to 1 | Hurt |
| raise model temperature T (higher entropy) | Importance sampling · Mobile Apps, tau(.)=4 | median RAE: error goes to 0 | Improved |
| use proposal-distribution importance sampling instead of naive Monte Carlo | K-step marginal estimation · all datasets, 4<=K<=15 | relative efficiency (variance ratio) | Improved |
| increase query horizon K | Importance sampling proposal · all datasets, 4<=K<=15 | relative efficiency | Hurt |
| increase query horizon K | all estimators · four LSTM datasets | median RAE | Hurt |
| use coverage-based beam search | beam search · query estimation | computation cost | Hurt |
Closest papers by meaning
- Future Language Modeling from Temporal Document History (ICLR 2024 · similarity 0.85)
- Conformal Autoregressive Generation: Beam Search with Coverage Guarantees (AAAI 2024 · similarity 0.84)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.84)
- A Neural Network Approach for Efficiently Answering Most Probable Explanation Queries in Probabilistic Models (NeurIPS 2024 · similarity 0.84)
- BARNN: A Bayesian Autoregressive and Recurrent Neural Network (ICML 2025 · similarity 0.84)
- Architectural and Inferential Inductive Biases for Exchangeable Sequence Modeling (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
