Research map / Large language models / Language, safety and interpretability
Eliciting Numerical Predictive Distributions of LLMs Without Auto-Regression
Julianna Piskorz, Kasia Kobalczyk, Mihaela van der Schaar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use magnitude-factorised probe instead of vanilla MLP probe with log-scaling (ablation) | regression probe · Synthetic scale 1.0, greedy target | MSE vs LLM statistic: 0.0400 -> 0.0150 | Improved |
| use magnitude-factorised probe instead of vanilla MLP probe (no log-scaling) (ablation) | regression probe · Synthetic scale 1.0, mean target | MSE vs LLM statistic: 0.0092 -> 0.0061 | Improved |
| two-phase training of order and value heads instead of joint training (ablation) | magnitude-factorised point probe · Synthetic data | MSE | Improved |
| two-phase training instead of joint training for quantile probe (ablation) | quantile probe · Synthetic data | quantile accuracy | No clear effect |
| concatenate hidden states from multiple layers instead of a single layer (ablation) | probe input · Synthetic data, Llama-2-7B | probe error | Improved |
| train probe on narrow context length range [10, 20] instead of [3, 40] (ablation) | quantile probe · Unseen context lengths, scale 1.0 | coverage deviation | Hurt |
| train probe on real data from all sub-datasets versus holding out sub-datasets (ablation) | quantile probe · Real-world data, 90% interval coverage | empirical coverage (%): 82.1 -> 88.5 | Improved |
| train probe on synthetic data only and test on real data (ablation) | quantile probe · Real-world data, 90% interval coverage | empirical coverage (%): 88.5 -> 67.7 | Hurt |
| use probe in place of 20-25 LLM samples for next-value prediction | magnitude-factorised probe · Synthetic scale 1.0 | MSE vs ground truth: probe outperforms empirical sampling for N up to 20-25 samples | Improved |
| use probe instead of autoregressive sampling for one prediction | Llama-2-7B · Inference time | time (s): 1.59 -> 0.034 | Improved |
| probe predicts greedy output instead of mean | magnitude-factorised probe · Synthetic scale 1.0 | MSE vs LLM statistic: 0.006 -> 0.015 | Hurt |
| probe prediction versus LLM's own accuracy on next value | probe · Synthetic scale 1.0 | MSE vs ground truth: 0.0562 vs 0.0555 | No clear effect |
Closest papers by meaning
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.91)
- LLM Flow Processes for Text-Conditioned Regression (NeurIPS 2026 · similarity 0.88)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.87)
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.87)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.87)
- Can LLMs Understand Time Series Anomalies? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
