Research map / 3D vision / 3D generation

GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

ICLR 2024 · 3D vision › 3D generation · approach: Semantic scene completion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Geometric Transform Attention (GTA), which applies group-element representations of camera pose and image position to queries, keys and values so attention is computed in an aligned coordinate frame, improving sparse wide-baseline novel view synthesis transformers without extra learned parameters.

Problem

Existing positional encodings (absolute, relative, and camera-embedding schemes) were designed for NLP or 2D sequences and do not respect the Euclidean geometric structure of multi-view 3D vision inputs, limiting learning efficiency in sparse wide-baseline novel view synthesis.

Approach

Encode each token's geometric attribute (camera extrinsic in SE(3) plus 2D image position in SO(2)xSO(2)) as a group representation matrix and apply the relative transformation directly to query, key and value features inside attention, so the attention output is computed in aligned local coordinates; this replaces additive or bias-based positional encodings and adds no learned parameters.

Key results

• On CLEVR-TR, GTA reaches 39.63 PSNR versus 37.27 for RePAST and 33.51 for SRT (test metrics table).
• On MSN-Hard, GTA reaches the same performance as RePAST using only 1/6 of the training steps.
• On DiT-B/2 ImageNet 256x256, DiT+GTA improves FID-50K from 7.02 to 5.87 and IS from 145.3 to 158.2.

Measured effects reported in the paper

Surprising findings

• Applying the transformation to values is important: removing it from V drops CLEVR-TR PSNR from 38.99 to 36.54.
• SRT performance depends heavily on the choice of reference coordinate system, whereas GTA is invariant by construction.

Limitations

• Requires known camera poses from external methods such as COLMAP.
• Additional computation overhead in encoding and rendering compared to SRT.

Experimental setup

Datasets: CLEVR-TR MSN-Hard RealEstate10k ACID ImageNet

Benchmarks: CLEVR-TR MSN-Hard RealEstate10k ACID ImageNet 256x256

Metrics: PSNR SSIM LPIPS IS FID-50K PR-AUC

Baselines: APE RPE SRT RePAST MLN ElemMul GBT FM RoPE+FTL DiT

Builds on: SRT RePAST RoPE Scene representation transformer DiT

Closest papers by meaning

• DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers (NeurIPS 2026 · similarity 0.88)
• Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis (NeurIPS 2026 · similarity 0.88)
• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.88)
• Cameras as Relative Positional Encoding (NeurIPS 2025 · similarity 0.87)
• Understanding Multi-View Transformers (NeurIPS 2026 · similarity 0.87)
• Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers (NeurIPS 2026 · similarity 0.87)