Research map / Architectures and efficiency / Attention and transformers
Geometry Matters in Packed 3D Attention
Amil Khan, Anantajit Subrahmanya, B.S. Manjunath
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Pack3D, a geometry-aware local-global attention operator for packed high-token 3D fields that separates memory layout from object geometry.
Problem
Attention over packed 3D volumes treats token sequence order as geometry, so sequence-index positional mechanisms can learn memory layout rather than the underlying object.
Approach
A geometry-aware local-global attention operator that separates memory layout from geometric structure, so positional information follows 3D geometry rather than packed sequence index.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use axial grid-coordinate RoPE instead of packed-sequence RoPE (ablation) | Pack3D (262k/depth1, 3.06M params) · Allen 6 structures | best macro Dice: 0.7142 -> 0.7375 | Improved |
| use axial grid-coordinate RoPE instead of packed-sequence RoPE (ablation) | Pack3D (65k/depth1) · Allen 6 structures | best macro Dice: 0.7099 -> 0.7346 | Improved |
| add coordinate-aware compressed branch to local-only model (ablation) | Pack3D (262k/depth8) · Allen 6 structures | best macro Dice: 0.8387 -> 0.8531 | Improved |
| use Pack3D instead of parameter-matched TokenGrid 3D U-Net | 3D U-Net · Allen-15 joint task | best macro Dice: 0.8134 -> 0.8228 | Improved |
| use Pack3D instead of U-Net per structure (final checkpoint) | 3D U-Net · Allen 6 structures, per structure final Dice | final Dice | Mixed |
| use summary-center RoPE instead of token-before-compression (ablation) | Pack3D · Allen 6 structures, 262k/depth1 | best Dice: 0.7378 -> 0.7384 | No clear effect |
| add RoPE to compressed branch (ablation) | Pack3D · Allen 6 structures, 262k/depth1 | best Dice: 0.7333 -> 0.7378 | Improved |
| normalize coordinates or use z-heavy RoPE allocation (ablation) | Pack3D · Allen 6 structures, 262k/depth1 | best Dice | Hurt |
| use z-light RoPE allocation instead of balanced (ablation) | Pack3D · Allen 6 structures, 262k/depth1 | best / final Dice: best 0.7378 -> 0.7384; final 0.7041 -> 0.6789 | Mixed |
| use learned separable 3D PE instead of axial RoPE (ablation) | Pack3D · Allen 6 structures, 262k/depth1 | final Dice: 0.7041 -> 0.7151 | Mixed |
| use additive 3D relative bias or ALiBi-style bias instead of axial RoPE (ablation) | Pack3D · Allen 6 structures, 65k/depth1 | best macro Dice: 0.7346 -> 0.7055 / 0.7098 | Hurt |
| corrupt compressed summaries (shuffle) at inference (ablation) | trained Pack3D · Allen 6 structures, 262k/depth8 | macro Dice: 0.8531 -> 0.6858 | Hurt |
Surprising findings
- Sequence-index positional mechanisms can learn the memory layout rather than the object when volumes are packed into tokens.
Closest papers by meaning
- Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging (CVPR 2026 · similarity 0.86)
- GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers (ICLR 2024 · similarity 0.86)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.85)
- FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry (NeurIPS 2026 · similarity 0.85)
- Riemann-based Multi-scale Attention Reasoning Network for Text-3D Retrieval (AAAI 2025 · similarity 0.85)
- Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
