Research map / Large language models / Language, safety and interpretability
Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HVM, a non-parametric hierarchical chunking-plus-variable learning model that builds abstract variables from sequences, compared against HCM, LZ78, associative learning, and LLMs on compression and a human sequence-recall transfer experiment.
Problem
Sequence learning models lack abstraction, causing memory inefficiency and poor transfer of abstract variables to new sequences; the paper asks how to learn and transfer abstract representations in a human-like way.
Approach
Extends hierarchical chunking (HCM) with variable discovery: chunks that appear in the same preceding/succeeding context are grouped into a variable (category), and a prefix-trie parsing graph organizes memory for faster chunk retrieval; abstraction is layered iteratively so nested structures emerge.
Key results
- On BabyLM domains, HVM achieves lower NLL than HCM across all four domains (e.g., CHILDES 1953 vs 2784) and a compression ratio of 0.36 on CHILDES vs 0.38 for LZ78.
- HVM sequence NLL correlates with human recall time in training (R=0.86, p<=0.001) and transfer blocks (R=0.7, p<=0.001).
- LLMs (GPT-2, Llama 2, Llama 3) do not differentiate training sequences sharing variables from control sequences, whereas HVM and HCM do.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add variable learning to chunk learning (ablation) | HCM (forming HVM) · CHILDES, BNC, Gutenberg, and Open Subtitles | NLL: 2783.71 -> 1953.01; 3591.60 -> 3108.33; 3770.36 -> 3252.84; 4151.89 -> 3764.48 | Helped |
| Add variable learning to chunk learning (ablation) | HCM (forming HVM) · Novel transfer sequence in the memory experiment | NLL | Helped |
| Add variable learning to chunk learning (ablation) | HCM (forming HVM) · Novel transfer sequence across abstraction layers | NLL | Helped |
| Organize chunks in a prefix parsing graph (ablation) | HCM (forming HVM) · Generated sequences | Parsing search steps | Helped |
| Use HVM sequence NLL as a predictor of human recall time | HVM · Training block of the human sequence memory task | Correlation with human recall time: R = 0.86, p ≤ 0.001 | Helped |
| Use HVM sequence NLL as a predictor of human recall time | HVM · Transfer block of the human sequence memory task | Correlation with human recall time: R = 0.7, p ≤ 0.001 | Helped |
| Train on sequences sharing variables with the transfer block | LLMs and associative learning model · Transfer block of the memory recall experiment | NLL differentiation between variable and control training | No clear effect |
Surprising findings
- LZ78 achieves the best coding efficiency in BabyLM domains, but HVM achieves lower sequence negative log-likelihood and in some domains better compression ratio.
- LLMs' NLL correlates more strongly with human recall in the training block, but cognitive models correlate much more strongly with human transfer performance.
Limitations
- Variables can only be embedded between chunks and cannot appear at the beginning or end of sequences.
- Later-learned representations depend on earlier ones; cannot capture patterns with a variable number of chunks between elements.
- Computational efficiency not explicitly optimized.
Experimental setup
Datasets: BabyLM CHILDES BNC Gutenberg Open Subtitles synthetic hierarchical sequences from generative model human sequence memory experiment (112 participants)
Benchmarks: BabyLM text domains (CHILDES, BNC, Gutenberg, Open Subtitles) human sequence memory recall (transfer block)
Metrics: compression ratio negative log-likelihood coding efficiency parsing search steps R-squared against human recall time
Baselines: HCM LZ78 associative learning model GPT-2 Llama 2 Llama 3
Builds on: hierarchical chunking model (HCM) LZ78 chunking models of sequence learning
Closest papers by meaning
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.87)
- Concept-Guided Interpretability via Neural Chunking (NeurIPS 2025 · similarity 0.86)
- From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (ICML 2025 · similarity 0.86)
- In-Context Learning Dynamics with Random Binary Sequences (ICLR 2024 · similarity 0.86)
- Human-inspired Episodic Memory for Infinite Context LLMs (ICLR 2025 · similarity 0.85)
- Language Model Memory and Memory Models for Language (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.