Research map / Graph learning / Graph self-supervised learning
Causality-Induced Positional Encoding for Transformer-Based Representation Learning of Non-Sequential Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAPE, which learns a weighted causal DAG over non-sequential features via a nonlinear SEM, embeds it in hyperbolic space, and converts the resulting positional encodings into rotary form for transformer self-attention.
Problem
Positional encodings assume a predefined token/feature order, so they fail for non-sequential but causally related features such as multi-omics gene or protein measurements.
Approach
Derive positional encodings from a learned causal DAG: causal strength maps to hyperbolic distance and causal specificity to distance from the origin, and these are converted to rotary form (a causality-aware RoPE) so attention attenuates with causal distance and generality.
Key results
- With scGPT, CAPE reduces MSE from 0.202 to 0.182 (single-gene) and 0.201 to 0.176 (double-gene perturbations).
- With scBERT, CAPE reduces MSE from 0.224 to 0.193 (single-gene) and 0.230 to 0.189 (double-gene).
- Average MSE reduction across both models is reported as 11.1%, versus 2.7% for a trainable relative position encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace positional encoding with CAPE | scBERT · Single-gene perturbation | MSE: 0.224 -> 0.193 | Helped |
| Replace positional encoding with CAPE | scBERT · Double-gene perturbations | MSE: 0.230 -> 0.189 | Helped |
| Replace positional encoding with CAPE | scGPT · Single-gene perturbation | MSE: 0.202 -> 0.182 | Helped |
| Replace positional encoding with CAPE | scGPT · Double-gene perturbations | MSE: 0.201 -> 0.176 | Helped |
| Add CAPE instead of omitting it (ablation) | scGPT · Single-gene perturbation | MSE: 0.234 -> 0.182 | Helped |
| Add CAPE instead of omitting it (ablation) | scGPT · Double-gene perturbations | MSE: 0.238 -> 0.176 | Helped |
| Use learned causal structure instead of a feature-correlation similarity graph (ablation) | CAPE with scGPT · Single-gene perturbation | MSE: 0.209 -> 0.182 | Helped |
| Use learned causal structure instead of a feature-correlation similarity graph (ablation) | CAPE with scGPT · Double-gene perturbations | MSE: 0.213 -> 0.176 | Helped |
| Use hyperbolic distance and RSGD instead of Euclidean distance and SGD (ablation) | CAPE with scGPT · Single-gene perturbation | MSE: 0.192 -> 0.182 | Helped |
| Use hyperbolic distance and RSGD instead of Euclidean distance and SGD (ablation) | CAPE with scGPT · Double-gene perturbations | MSE: 0.196 -> 0.176 | Helped |
| Convert hyperbolic positional encodings to rotary form instead of adding them directly (ablation) | CAPE with scGPT · Single-gene perturbation | MSE: 0.201 -> 0.182 | Helped |
| Convert hyperbolic positional encodings to rotary form instead of adding them directly (ablation) | CAPE with scGPT · Double-gene perturbations | MSE: 0.208 -> 0.176 | Helped |
Limitations
- Evaluation limited to a single real perturbation dataset for the main downstream task
- Relies on a simplified SEM with Gaussian exogenous noise and MLP f for causal discovery
- Causal graph quality depends on accurate causal structure learning from observational data
Experimental setup
Datasets: synthetic tabular data (gCastle IIDSimulation, 5000x10) human leukemia scRNA-seq perturbation dataset
Benchmarks: gene perturbation prediction on human leukemia scRNA-seq (single- and double-gene perturbations)
Metrics: mean squared error on top 20 differentially expressed genes
Baselines: scBERT static absolute positional encoding scGPT trainable absolute positional encoding trainable relative positional encoding
Builds on: RoPE NOTEARS acyclicity constraint scGPT scBERT GEARS
Closest papers by meaning
- A Fixed-Point Approach for Causal Generative Modeling (ICML 2024 · similarity 0.87)
- How Transformers Learn Causal Structure with Gradient Descent (ICML 2024 · similarity 0.87)
- Learnable Spatial-Temporal Positional Encoding for Link Prediction (ICML 2025 · similarity 0.86)
- Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.86)
- Selective Induction Heads: How Transformers Select Causal Structures in Context (ICLR 2025 · similarity 0.86)
- StableMask: Refining Causal Masking in Decoder-only Transformer (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.