Research map / Time series / Foundation models for time series
Density estimation with LLMs: a geometric investigation of in-context learning trajectories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes LLMs' in-context density estimation by serializing samples as 2-digit numbers, extracting PDFs with Hierarchy-PDF, and visualizing trajectories with InPCA, finding LLM trajectories resemble an adaptive-kernel KDE with two parameters.
Problem
How LLMs estimate probability density functions from data observed in context, and what geometric/algorithmic process underlies this in-context density estimation compared with classical methods like histograms and KDE.
Approach
Uses InPCA on Hellinger distances to embed in-context density estimation trajectories of pretrained LLMs alongside histogram and KDE trajectories, then fits a two-parameter bespoke KDE with adaptive kernel shape and bandwidth that reproduces LLM behavior.
Key results
- Gaussian targets: 2D InPCA embeddings capture about 92% of pairwise Hellinger distances; LLaMA-2 trajectories lie between the geodesic and the Gaussian submanifold, similar across LLaMA-2, Gemma, and Mistral.
- A two-parameter bespoke KDE with adaptive shape and bandwidth closely imitates LLaMA-2 13B in-context DE trajectories, with fitted shape s evolving from about 0.1 to about 1.
- The fitted LLM kernel width decays faster than the standard n^(-1/5) KDE schedule, producing a longer trajectory in InPCA space.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMA-2 in-context density estimation instead of Gaussian KDE | Density estimation on narrow Gaussian targets · Narrow Gaussian targets | qualitative convergence to the target PDF | Helped |
| Use LLaMA-based density estimation instead of Gaussian KDE | Density estimation on narrow uniform targets · Narrow uniform targets | qualitative ability to reach the target PDF | Helped |
| Use a fitted adaptive kernel shape and bandwidth | Bespoke KDE · LLaMA-2 13B density-estimation trajectories | trajectory similarity | Helped |
| Replace the standard n^{-1/5} bandwidth schedule with a faster-decaying fitted schedule | KDE on a narrow Gaussian target · Narrow Gaussian target | trajectory similarity to LLaMA-2 13B | Helped |
| Vary kernel shape from tophat and Gaussian toward smaller shape parameters (ablation) | KDE on a narrow Gaussian target · Narrow Gaussian target | trajectory similarity to LLaMA-13B | Helped |
| Fit bespoke KDE to Bayesian histogram trajectories | Bespoke KDE · Bayesian histogram trajectories | trajectory similarity | Hurt |
| Use standard PCA instead of InPCA (ablation) | Visualization of LLaMA-2 density-estimation trajectories · LLaMA-2 trajectories | geometric clarity of trajectories: 91% vs 83% | Hurt |
Surprising findings
- Bayesian histograms cannot be faithfully imitated by the bespoke KDE model despite superficial similarity to a tophat KDE.
- LLM in-context density estimation is more adaptive than classical KDE, with the LLM departing from the Gaussian submanifold for narrow uniform targets where KDE fails.
Limitations
- Analysis restricted to context length 200 and 2-digit numbers over a fixed support (0,100).
- Bespoke KDE does not fit the Bayesian histogram and fits LLaMA-2 only approximately; Gemma-2b and Mistral are approximated but not identically.
- Mechanistic link to dispersive induction heads is speculative and not tested.
Experimental setup
Datasets: synthetic samples from Gaussian, uniform, and Gaussian-process-generated PDFs
Metrics: Hellinger distance pairwise distance variance captured by InPCA embedding KL-divergence for bespoke KDE fitting
Baselines: Gaussian KDE Bayesian histogram
Builds on: Hierarchy-PDF InPCA Gaussian kernel density estimation induction heads
Closest papers by meaning
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.87)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.86)
- Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective (ICML 2024 · similarity 0.86)
- Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.86)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.85)
- In-Context Learning Dynamics with Random Binary Sequences (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.