Research map / Large language models / Language, safety and interpretability
Are Language Models Any Good at Density Modeling?
Sriram Ranga, Sai Shashank Bedampeta, Rui Mao, Anupam Chattopadhyay
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on uniform distribution over numbers | GPT-2-style LM · Uniform | KL divergence: 0.2282 -> 0.0038 | Improved |
| train on linear distribution | GPT-2-style LM · Linear | KL divergence: 0.3874 -> 0.0290 | Improved |
| train on binomial distribution | GPT-2-style LM · Binomial | KL divergence: 82.0749 -> 24.8572 | Improved |
| train on sine distribution with period 10 (syntactic suffix cue) | GPT-2-style LM · Sine period 10 | KL divergence: 0.4274 -> 0.0070 | Improved |
| train on sine distribution with period 3 (needs semantic divisibility) | GPT-2-style LM · Sine period 3 | KL divergence: 0.6880 -> 0.7323 | Hurt |
| train on divisibility-by-3 distribution | GPT-2-style LM · Divisibility by 3 pattern | distribution fit | Hurt |
| increase fraction of language used for training while keeping steps fixed | GPT-2-style LM · Syntax error rate | incorrectly formed numbers | No clear effect |
| train on 10% of the numbers instead of more data | GPT-2-style LM · Syntax error rate | incorrectly formed numbers: less than 1% | No clear effect |
| numbers frequently appearing as substrings of other numbers | GPT-2-style LM trained on uniform · Probability of numbers one to one hundred | assigned probability | Hurt |
| train in passage mode with multiple numbers per passage | GPT-2-style LM · First-number distribution | distribution fit | No clear effect |
Closest papers by meaning
- Are Large Language Models Good Statisticians? (NeurIPS 2024 · similarity 0.87)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.87)
- When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.87)
- LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language (NeurIPS 2024 · similarity 0.87)
- MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models (ICML 2025 · similarity 0.86)
- Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
