Research map / Large language models / Language, safety and interpretability
From Tokens to Lattices: Emergent Lattice Structures in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that masked language models implicitly learn a probabilistic formal context whose conditional probabilities can be turned into concept lattices via Formal Concept Analysis, without human-defined concepts.
Problem
How conceptual (lattice) structure emerges from masked language model pretraining, and whether concept lattices can be recovered from MLMs without relying on human-defined concepts or ontologies.
Approach
Interprets MLM conditional probabilities between objects and attributes under cloze patterns as a probabilistic triadic formal context, then builds concept lattices from it via FCA (with multi-pattern pooling and min-max binarization), and grounds the analysis in a distributional view of concept learning.
Key results
- On Region-language, BertLattice (max.) with BERT-base reaches MRR 0.842 and Hit@1 0.751, versus 0.340 MRR for the BertEmb. baseline.
- On Disease-symptom concept classification, BioBERT BertLattice (max.) reaches F1 0.611 and mAP 0.389, versus 0.345 F1 for BertEmb.
- Lattice reconstruction on Region-language surfaces latent concepts such as 'German and French speaking regions' not predefined by humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conditional-probability formal contexts instead of hidden-state distance embeddings | BERT-base on Region-language · Region-language | MRR: 0.340 -> 0.842 | Helped |
| Use conditional-probability formal contexts instead of hidden-state distance embeddings | BERT-base on Animal-behavior · Animal-behavior | MRR | Helped |
| Use conditional-probability formal contexts instead of hidden-state distance embeddings | BERT-large on Region-language · Region-language | F1: 0.231 -> 0.709 | Helped |
| Use conditional-probability formal contexts instead of hidden-state distance embeddings | BioBERT on Disease-symptom · Disease-symptom | F1: 0.345 -> 0.611 | Helped |
| Use BioBERT pretrained on biomedical texts rather than BERT-base | BertLattice (max.) · Disease-symptom | F1 | Helped |
| Switch from average pooling to max pooling (ablation) | BertLattice · Region-language | MRR: 0.823 -> 0.842 | Helped |
| Switch from average pooling to max pooling (ablation) | BertLattice · Animal-behavior | Hit@10: 0.537 -> 0.519 | Mixed |
| Use sigmoid normalization instead of min-max normalization (ablation) | BertLattice lattice construction · lattice construction | performance | Mixed |
| Raise the threshold under min-max normalization (ablation) | BertLattice lattice construction · lattice construction | performance | No clear effect |
Surprising findings
- Pretrained BERT-distill and BERT-large do not uniformly beat BERT-base; BERT-base is best in MRR and Hit@1 while others win on higher Hit@k.
Limitations
- Framework limited to single-relational data.
- Requires either objects or attributes to be single vocabulary tokens for the efficient construction.
- Performance on domain-specific disease-symptom data is weaker for general BERT models, reflecting pretraining coverage.
Experimental setup
Datasets: Wiki44k Wikidata official-language relation Kaggle disease-symptom dataset human-curated animal-behavior list
Benchmarks: Region-language Animal-behavior Disease-symptom
Metrics: MRR Hit@1 Hit@5 Hit@10 F1 mAP
Baselines: BertEmb.
Builds on: formal concept analysis triadic formal context masked language modeling Gaussian graphical models
Closest papers by meaning
- Latent Concept Disentanglement in Transformer-based Language Models (ICLR 2026 · similarity 0.86)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.85)
- ICLR: In-Context Learning of Representations (ICLR 2025 · similarity 0.85)
- On the Origins of Linear Representations in Large Language Models (ICML 2024 · similarity 0.85)
- From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When (NeurIPS 2024 · similarity 0.85)
- The Geometry of Categorical and Hierarchical Concepts in Large Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.