Research map / Vision-language and multimodal / Human and embodied modeling
Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes relational learning in pre-trained models as recovery of a weighted world hypergraph from hyperedge samples, proves identifiability and a near-optimal sample complexity for masked modeling, and extends the framework to multimodal entity alignment.
Problem
How pre-trained foundation models acquire relational understanding of entities; what data is needed to recover the relational structure of the world, framed as hypergraph recovery.
Approach
Models the world as a weighted hypergraph (entities as nodes, relations as hyperedges) with pre-training data as i.i.d. hyperedge samples, and shows masked modeling learns relative hyperedge weights along masked-connection paths, giving near-optimal sample complexity Õ(m/ε²) matching an Ω(m/ε²) lower bound.
Key results
- Masked modeling achieves near-optimal sample complexity Õ(m/ε²) matching the Ω(m/ε²) information-theoretic lower bound up to log factors.
- In real-world relation evaluation against ConceptNet subgraphs, GPT-4 achieves the best overall normalized L1 distance among evaluated LLMs, with GPT-3.5 slightly better than LLaMA-2-70B.
- Synthetic experiments show reconstruction error decreases with training, and more edges and longer MM path lengths require more steps to converge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Larger number of edges (m) in STAR graphs (ablation) | BERT masked modeling on synthetic graphs · Synthetic relational learning | convergence steps: more steps to converge | Hurt |
Surprising findings
- The effect of weight range ratio on convergence is not obvious in experiments, suggesting a gap between the theoretical bound and practice.
- Entity alignment is theoretically feasible without labeled pairs, but labeled pairs are practically needed to reduce computational complexity.
Limitations
- Simplified mathematical model; perception mapping assumed bijective and data i.i.d. from hyperedges.
- Real-world evaluation uses only unweighted graphs because LLM relation weights are hard to elicit.
- Entity alignment in practice requires labeled pairs due to graph isomorphism computational difficulty; multimodal extension only illustrated.
Experimental setup
Datasets: ConceptNet synthetic graph-generated text samples
Benchmarks: ConceptNet subgraphs
Metrics: normalized L1 distance between adjacency matrices reconstruction error of unweighted and weighted graphs
Baselines: Llama-2-70B GPT-3.5 GPT-4
Builds on: masked language modeling star expansion Hoeffding inequality
Closest papers by meaning
- SPHINX: Structural Prediction using Hypergraph Inference Network (ICML 2025 · similarity 0.87)
- Deep Representation Learning for Forecasting Recursive and Multi-Relational Events in Temporal Networks (AAAI 2025 · similarity 0.87)
- Neural Temporal Point Processes for Forecasting Directional Relations in Evolving Hypergraphs (AAAI 2025 · similarity 0.87)
- Hypergraph Representation Learning with Hyperlink Random Effects (NeurIPS 2026 · similarity 0.86)
- Toward a Unified Statistical Theory of Unsupervised Pretraining and Supervised Neural Knowledge Graph Learning (NeurIPS 2026 · similarity 0.86)
- Hypergraph-enhanced Dual Semi-supervised Graph Classification (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.