Research map / Label-efficient and robust learning / Recognition and long-tail
A Pre-training Framework for Relational Data with Information-theoretic Principles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task Vector Estimation (TVE), a pre-training framework for relational deep learning that regresses set-based SQL-style aggregates of next-window k-hop neighbor values, combined optionally with MAE or contrastive losses, and evaluates on RelBench tasks.
Problem
Pre-training for relational databases is hard because downstream tasks are heterogeneous (schema graph, temporal windows, SQL-defined labels), and generic SSL (masked reconstruction, contrastive learning) can capture noise unrelated to labels.
Approach
Construct pretext targets as task vectors: set-level aggregate statistics (mean, count, etc.) over k-hop schema-traversal neighbors within the next time window, and train the RDL encoder to estimate them with a scaled cosine error; justified by an information-theoretic argument that side-channel task-aware representations retain at least as much label information.
Key results
- In the low-data regime, TVE-2 achieves the best test result on 5 of 6 classification tasks (e.g., rel-amz/user-churn top50 test ROC-AUC 93.21 for TVE-1 vs 89.07 baseline).
- In data-sufficient RelBench tasks, gains from TVE alone are marginal; hybrid CTR+TVE and MAE+TVE variants outperform single-objective models on five of six tasks.
- Linear probing shows TVE outperforms MAE, CTR and the baseline on every evaluated task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TVE pre-training instead of standard pre-training objectives | RelBench data-limited tasks · Low-data classification and regression tasks | ROC-AUC / MAE | Mixed |
| Add CTR loss to TVE pre-training (ablation) | TVE on data-sufficient RelBench tasks · rel-amz/item-churn, rel-amz/user-churn, rel-hm/user-churn, rel-amz/item-ltv, rel-amz/user-ltv | ROC-AUC / MAE | Helped |
| Add MAE loss to TVE pre-training (ablation) | TVE on data-sufficient RelBench tasks · rel-amz/item-churn, rel-amz/user-churn, rel-hm/user-churn, rel-amz/item-ltv, rel-amz/user-ltv, rel-hm/item-sales | ROC-AUC / MAE | Mixed |
| Use 2-hop rather than 1-hop schema traversal (ablation) | TVE · Data-limited tasks | ROC-AUC / MAE | Mixed |
| Use CTR-0.25 rather than CTR-0.5 (ablation) | Graph Contrastive Learning · rel-amz/item-churn least50 test | ROC-AUC: 79.89 ± 4.41 -> 83.97 ± 2.72 | Helped |
| Use MAE-0.25 pre-training (ablation) | RelBench models · Data-sufficient tasks | ROC-AUC / MAE | Mixed |
Surprising findings
- Contrastive learning with 0.25 masking reached 83.97 test ROC-AUC on rel-amz/item-churn least50 but fell to 79.89 at 0.5 masking, showing flaky sensitivity.
- In data-sufficient settings, MAE-0.25 alone beats the baseline on rel-amz/item-churn, and pre-training gains are often negligible or negative.
Limitations
- Task vectors built from simple SQL aggregates may not capture the full next-window set.
- Task vector dimensionality grows with columns and join paths.
- Evaluated only on rel-amazon and rel-hm node-level tasks.
Experimental setup
Datasets: rel-amazon rel-hm
Benchmarks: RelBench rel-amazon tasks RelBench rel-hm tasks new data-limited tasks
Metrics: ROC-AUC MAE
Baselines: Baseline (no pre-training) GraphMAE-style MAE (0.25, 0.5 mask) GraphCL-style contrastive learning (0.25, 0.5 mask)
Builds on: GraphMAE GraphCL RelBench Relational Deep Learning SCARF
Closest papers by meaning
- Griffin: Towards a Graph-Centric Relational Database Foundation Model (ICML 2025 · similarity 0.86)
- Knowledge Composition using Task Vectors with Learned Anisotropic Scaling (NeurIPS 2024 · similarity 0.84)
- Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective (ICML 2024 · similarity 0.84)
- Position: Relational Deep Learning - Graph Representation Learning on Relational Databases (ICML 2024 · similarity 0.84)
- Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models (ICML 2024 · similarity 0.83)
- UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.