Research map / Large language models / Language, safety and interpretability
LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LLM Processes, which elicit coherent numerical predictive distributions from pretrained LLMs via prompting (no weight updates) and conditions them on both numerical data and natural-language text for zero-shot regression, forecasting, and optimization.
Problem
Integrating prior knowledge expressed in natural language into probabilistic numerical prediction, and eliciting coherent joint predictive distributions from LLMs for multi-dimensional regression and density estimation.
Approach
Elicit continuous predictive densities from LLM digit-level token probabilities (hierarchical softmax over bins with a terminal token, original scale kept), and define joint distributions either as independent marginals (I-LLMP) or autoregressively (A-LLMP) by placing training and target points in the prompt; text conditioning is added to the prompt prefix.
Key results
- Mixtral-8x7B A-LLMP achieves lower NLL than a GP with RBF kernel on 10 of 12 synthetic functions (averaged over 7 sizes and 3 seeds).
- On the weather dataset, A-LLMP beats LLMTime (Llama-2-7B) in MAE and NLL, with the margin growing as training data is removed.
- On From Words to Numbers' Original #1 task, I-LLMP average MAE was 0.836 versus 3.137 for the prior method (Llama-2-7B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A-LLMP instead of a GP with an RBF kernel | Mixtral-8×7B regression · Function Dataset (12 synthetic functions) | NLL | Helped |
| Order training points by distance to the target rather than randomly or sequentially (ablation) | I-LLMP and A-LLMP · Synthetic regression | NLL | Helped |
| Use autoregressive predictions rather than independent marginal predictions (ablation) | LLMP · Synthetic regression | log-likelihood | Helped |
| Order autoregressive target points by distance to the closest training point (ablation) | A-LLMP · Synthetic regression | log-likelihood | Helped |
| Use the , \n prompt format rather than formats with less or more delimiters (ablation) | LLMP prompt · Sigmoid, Quadratic, and Linear + Cosine | NLL | Helped |
| Add labeled feature names and descriptions to numerical features (ablation) | Mixtral-8×7B Instruct I-LLMP · US housing prices | MAE | Helped |
| Add unlabelled raw feature values (ablation) | Mixtral-8×7B Instruct I-LLMP · US housing prices | MAE | No clear effect |
| Add in-context historical examples (ablation) | Mixtral-8×7B A-LLMP · Canadian monthly precipitation forecasting | NLL | Mixed |
| Scale the y-values across a wider range or include negative values (ablation) | I-LLMP prompt · Synthetic regression | NLL | Hurt |
| Vary top-p or softmax temperature (ablation) | LLMP sampling · Synthetic regression | NLL | No clear effect |
| Condition on scenario-relevant natural-language text | Llama-3-70B A-LLMP · Scenario-conditional predictions | predictive performance | Helped |
| Use distance-sorted training examples and a distributional median estimate rather than unsorted single-point generation | Llama-2-7B I-LLMP · From Words to Numbers Original #1 | MAE: 3.137 -> 0.836 | Helped |
Surprising findings
- Performance is surprisingly insensitive to top-p and softmax temperature.
- Distance-ordering target points gives performance competitive with or better than random ordering in autoregressive mode.
Limitations
- Maximum context size limits task size and amount of text conditioning.
- Much higher computational cost than GPs and standard regressors.
- Only smaller open-source LLMs were evaluated, not proprietary frontier models.
Experimental setup
Datasets: Function Dataset (12 synthetic 1D functions) Weather dataset (London daily temperature, precipitation, wind speed) Fashion-MNIST Canadian monthly precipitation (13 locations) US housing prices dataset (39980 listings) Singapore and San Diego monthly precipitation
Benchmarks: Function Dataset Weather dataset Fashion-MNIST reconstruction Black-box optimization six functions (Optuna comparison) Canadian precipitation in-context learning US housing prices
Metrics: NLL MAE
Baselines: Gaussian Process (RBF kernel) LLMTime Optuna From Words to Numbers multi-output GP
Builds on: LLMTime Neural Processes From Words to Numbers
Closest papers by meaning
- LLM Flow Processes for Text-Conditioned Regression (NeurIPS 2026 · similarity 0.89)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.88)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.88)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- Can LLMs Understand Time Series Anomalies? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.