Research map / Large language models / Retrieval and knowledge
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs finetuned only on indirect observations of a latent (e.g., distances to unknown city, coin flips, input-output pairs) can verbalize and apply that latent downstream without in-context examples or chain of thought.
Problem
Whether LLMs can infer latent information aggregated across scattered training documents and use it out-of-context, which bears on whether censoring dangerous knowledge from training data is effective and on monitoring what models learn.
Approach
Introduces inductive out-of-context reasoning (OOCR): finetune on documents containing only implicit evidence of a latent variable, then test on out-of-distribution downstream queries (including reflection questions) with no in-context examples or chain of thought, and compare against shuffled-prompt baselines and in-context learning.
Key results
- Finetuned GPT-3.5 Locations models answer the unknown city's identity correctly 56% of the time on average in free-form reflection, and assign ~57% probability to the correct food under closest-country distractors.
- Finetuned GPT-3.5 models on Coins correctly classified 22 of 24 finetunes by bias (0.7 vs 0.8); GPT-4 classified 8 of 8.
- OOCR outperformed in-context learning across tasks, and GPT-4 outperformed GPT-3.5 on OOCR evaluations for 4 of 5 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on implicit distance and direction observations rather than leave the model untrained | GPT-3.5 · Locations, city reflection | city identification accuracy: 56% | Helped |
| fine-tune on implicit location observations and answer food questions requiring country inference | GPT-3.5 · Locations, food with neighboring-country distractors | probability assigned to correct food: 57% | Helped |
| fine-tune on parity observations to infer variable assignments | GPT-3.5 · Parity Learning | probability of correct variable values: 80% | Helped |
| fine-tune on implicit training documents instead of presenting them in context | GPT-3.5 · OOCR evaluations across tasks | average OOCR performance | Helped |
| use GPT-4 instead of GPT-3.5 | inductive OOCR · OOCR evaluations across tasks | average OOCR performance | Helped |
| use GPT-4 instead of GPT-3.5 | Locations · held-out training-task test set | distance prediction error: ≈280 to ≈180 | Helped |
| use GPT-4 instead of GPT-3.5 | inductive OOCR · Coins, Mixture of Functions, and Parity Learning training tasks | in-distribution test performance | No clear effect |
| train on coin bias 0.8 rather than 0.7 | GPT-3.5 and GPT-4 · Coins, distinguishing bias 0.7 from 0.8 | classification of fine-tunes from assigned probabilities: 22 out of 24 GPT-3.5 and 8 out of 8 GPT-4 | Helped |
Surprising findings
- Models finetuned only on distances to an unknown city can verbalize that it is Paris and answer food questions without chain of thought.
- In-context learning with up to 200 examples essentially failed on Locations, Coins, and Parity Learning, performing worse than finetuning.
Limitations
- Relies on API finetuning, so architecture, training data, and finetuning algorithm are unknown; GPT-3.5 vs GPT-4 comparison confounded.
- Uses specially constructed finetuning datasets rather than realistic pretraining data.
- OOCR performance is high variance and sensitive to prompts, especially on Mixture of Functions; absolute performance is unreliable.
Experimental setup
Datasets: GeoNames cities500
Benchmarks: Locations OOCR evaluations Functions reflection evaluations Mixture of Functions reflection evaluations Coins Parity Learning
Metrics: probability of correct answer median distance error (km) total variation distance bootstrapped 90% confidence intervals
Baselines: in-context learning untrained model shuffled-prompt baseline
Builds on: Reversal Curse in-context learning as gradient descent
Closest papers by meaning
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.88)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.