Research map / Large language models / Language, safety and interpretability
In-Context Learning Dynamics with Random Binary Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses random binary coin-flip sequences as a probe domain to show that GPT-3.5+ LLMs generate subjectively random sequences and exhibit sharp, model-selection-like phase transitions in in-context learning of formal languages, without accessing internal activations.
Problem
Understanding which latent concepts LLMs activate under different in-context prompts, and whether in-context learning changes smoothly (model averaging) or abruptly (model selection), using only black-box behavior.
Approach
Frames ICL as Bayesian model selection over latent concepts and uses random binary sequences (subjective randomness from cognitive science) as a minimal domain to study generation and judgment dynamics; compares GPT outputs to Bernoulli, Window Average, and Markov chain models and measures sharp S-shaped transitions in formal-language concept probabilities as context grows.
Key results
- GPT-3.5+ models (text-davinci-003, gpt-3.5-turbo, gpt-4) generate controllable binary sequences whose means track the specified p(Tails) and whose run-length statistics deviate from Bernoulli toward a Gambler's-fallacy-like avoidance of long runs; a Window Average model (w=5) partly explains this.
- In formal-language generation tasks, gpt-3.5-turbo-instruct-0914 transitions sharply from random-looking outputs to outputs matching concepts such as (011)^n as context length grows, with S-shaped curves rather than smooth loss decreases.
- In randomness judgment tasks across 9 binary concepts, text-davinci-003 shows sharp transitions from confidently "Random" to confidently "Non-random" at concept-dependent context lengths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate sequences with text-davinci-003 instead of sampling from a Bernoulli process with the same mean | Randomness Generation · Random binary sequences | sequence-mean variance | Mixed |
| Use the Window Average model instead of a Bernoulli process | Randomness Generation · Random binary sequences | sequence means and longest-run lengths | Helped |
| Increase context length with examples of a simple formal language | gpt-3.5-turbo-instruct-0914 · Formal Language Generation | probability mass assigned to valid continuations | Helped |
| Increase context length with a formal-language sequence | text-davinci-003 · Formal Language Generation | probability mass assigned to valid continuations | Mixed |
| Use shallower prediction depths (d < 4) | Formal Language Generation analysis · Formal languages | stability of in-context learning dynamics | Hurt |
| Append a repeating Heads-Tails distractor sequence to an MMLU prompt | text-davinci-003 · MMLU | probability of returning to the task answer | No clear effect |
| Use longer formal concepts | text-davinci-003 · Randomness Judgment | sharpness of the transition from Random to Non-random | Hurt |
| Use a fair-coin prompt instead of a prompt describing a fair coin or a non-random algorithm | GPT-3.5 · Formal Language Generation | formal concept learning dynamics | No clear effect |
| Generate sequences with GPT-3.5+ rather than a Bernoulli process | Randomness Generation · Sequence complexity | number of unique subsequences, Gzip file size, and inter-sequence Levenshtein di | Mixed |
Surprising findings
- With p(Tails)=50% exactly, GPT-3.5+ outputs are dominated by a repeating Heads, Tails pattern, producing a kink at 0.5 across metrics.
- Earlier GPT models such as text-davinci-001/002 are heavily biased and uncontrollable, and text-ada-001 shows S-shaped ICL dynamics while most mid-size models show none.
Limitations
- Relies on closed OpenAI API models whose versions change; text-davinci-003 is no longer publicly accessible as of January 2024.
- Breakdown of formal-language learning for longer concepts and differences across models are not fully explained.
- Analysis uses limited hypothesis space and approximations (depth-limited probability trees, comma-token omission).
Experimental setup
Datasets: MMLU GSM8K
Benchmarks: MMLU
Metrics: sequence mean distribution longest run length sub-sequence frequency gzip compressed size Levenshtein distance predictive probability of formal language concept
Baselines: Bernoulli process Window Average model Markov chain (order 3)
Builds on: Bayesian program induction subjective randomness models (Griffiths & Tenenbaum) Bayesian model selection view of ICL
Closest papers by meaning
- Probing the Decision Boundaries of In-context Learning in Large Language Models (NeurIPS 2024 · similarity 0.90)
- Competition Dynamics Shape Algorithmic Phases of In-Context Learning (ICLR 2025 · similarity 0.90)
- Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.88)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.88)
- The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains (NeurIPS 2024 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.