Research map / Architectures and efficiency / Attention and transformers
GeoAggregator: An Efficient Transformer Model for Geo-Spatial Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeoAggregator, a lightweight transformer for geospatial tabular regression using Gaussian-biased local attention, 2D rotary positional embedding, and Cartesian product attention (MCPA), evaluated against spatial statistical and deep geospatial baselines.
Problem
Deep learning models for geospatial tabular regression scale poorly with dataset size and lack explicit geographic priors for spatial autocorrelation and spatial heterogeneity.
Approach
A transformer encoder-processor-decoder over spatial neighborhoods, with a Gaussian distance bias added to attention logits, 2D rotary positional embeddings for locations, and a Cartesian-product attention that splits features into covariate and target streams to cut parameters and FLOPs, plus inducing points for linear complexity.
Key results
- GeoAggregator variants achieve seven best results out of 16 metric cells on synthetic datasets (per table caption), and GA-mini has the best MAE on Lin-r and the best MAE on SL-d-style comparisons among transformers.
- On Housing, GA-mini and Vanilla-mini reach the top MAE/R2 region (GA-mini R2 0.911), and on Poverty GA-small gets the lowest MAE (3.537).
- GA-mini uses about 4.3K parameters and 0.7M FLOPs on the synthetic data, far fewer than GCNNWR (1930K params, 474.6M FLOPs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GA-mini instead of the strongest listed baseline | GeoAggregator · Lin-r | MAE: 0.873 -> 0.870 | Helped |
| Use GA-mini instead of the strongest listed baseline | GeoAggregator · SL-r | MAE: 1.133 -> 1.269 | Hurt |
| Use GA-mini instead of the strongest listed baseline | GeoAggregator · SLX-r | MAE: 0.839 -> 0.881 | Hurt |
| Use GA-mini instead of the strongest listed baseline | GeoAggregator · Durbin-r | MAE: 0.897 -> 0.946 | Hurt |
| Use GA-mini instead of the strongest listed baseline | GeoAggregator · SLX-d | MAE: 0.807 -> 0.804 | Helped |
| Use GA-small instead of the strongest listed baseline | GeoAggregator · Poverty | MAE: 3.563 -> 3.537 | Helped |
| Replace MCPA with vanilla attention (ablation) | GeoAggregator · Housing | R²: 0.911 -> 0.906 | Hurt |
| Replace MCPA with vanilla attention (ablation) | GeoAggregator · Poverty | MAE: 3.537 vs 3.563 | Mixed |
| Increase input sequence length to ℓmax=1024 (ablation) | GA-mini without processor module · Housing | test MAE: 0.618 | Helped |
| Use a balanced attention bias factor λ instead of an extreme λ (ablation) | GeoAggregator · Housing | test performance | Helped |
Surprising findings
- Larger GeoAggregator models tend to perform worse than smaller ones, attributed to overfitting and training difficulty.
- On the PM25 dataset GeoAggregator underperforms the best baseline, attributed to long-range dependence contributing less under strong spatial heterogeneity.
Limitations
- Performance is inconsistent: underperforms on PM25 and larger variants overfit on some datasets.
- Does not yet handle categorical variables; spatial bias is a single shared lambda across features.
Experimental setup
Datasets: Synthetic spatial DGPs (Lin, SL, SLX, Durbin) with random and DEM covariates PM25 Housing (King County) Poverty (continental US)
Benchmarks: Synthetic datasets (8 DGP variants) PM25 Housing Poverty
Metrics: MAE R-squared #Params #FLOPs
Baselines: GWR XGBoost SRGCNN GCNNWR GSH-EL Vanilla attention GeoAggregator
Builds on: Geographically Weighted Regression Set Transformer inducing points rotary positional embedding
Closest papers by meaning
- TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning (NeurIPS 2024 · similarity 0.85)
- TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024 · similarity 0.83)
- Poly2Vec: Polymorphic Fourier-Based Encoding of Geospatial Objects for GeoAI Applications (ICML 2025 · similarity 0.83)
- GeoRouter: Dynamic Paradigm Routing for Worldwide Image Geolocalization (NeurIPS 2026 · similarity 0.83)
- Earthfarsser: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model (AAAI 2024 · similarity 0.83)
- GeoRanker: Distance-Aware Ranking for Worldwide Image Geolocalization (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.