AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Geometry Matters in Packed 3D Attention

Amil Khan, Anantajit Subrahmanya, B.S. Manjunath

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Block-sparse attention kernels · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Pack3D, a geometry-aware local-global attention operator for packed high-token 3D fields that separates memory layout from object geometry.

Problem

Attention over packed 3D volumes treats token sequence order as geometry, so sequence-index positional mechanisms can learn memory layout rather than the underlying object.

Approach

A geometry-aware local-global attention operator that separates memory layout from geometric structure, so positional information follows 3D geometry rather than packed sequence index.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use axial grid-coordinate RoPE instead of packed-sequence RoPE (ablation)Pack3D (262k/depth1, 3.06M params) · Allen 6 structuresbest macro Dice: 0.7142 -> 0.7375Improved
use axial grid-coordinate RoPE instead of packed-sequence RoPE (ablation)Pack3D (65k/depth1) · Allen 6 structuresbest macro Dice: 0.7099 -> 0.7346Improved
add coordinate-aware compressed branch to local-only model (ablation)Pack3D (262k/depth8) · Allen 6 structuresbest macro Dice: 0.8387 -> 0.8531Improved
use Pack3D instead of parameter-matched TokenGrid 3D U-Net3D U-Net · Allen-15 joint taskbest macro Dice: 0.8134 -> 0.8228Improved
use Pack3D instead of U-Net per structure (final checkpoint)3D U-Net · Allen 6 structures, per structure final Dicefinal DiceMixed
use summary-center RoPE instead of token-before-compression (ablation)Pack3D · Allen 6 structures, 262k/depth1best Dice: 0.7378 -> 0.7384No clear effect
add RoPE to compressed branch (ablation)Pack3D · Allen 6 structures, 262k/depth1best Dice: 0.7333 -> 0.7378Improved
normalize coordinates or use z-heavy RoPE allocation (ablation)Pack3D · Allen 6 structures, 262k/depth1best DiceHurt
use z-light RoPE allocation instead of balanced (ablation)Pack3D · Allen 6 structures, 262k/depth1best / final Dice: best 0.7378 -> 0.7384; final 0.7041 -> 0.6789Mixed
use learned separable 3D PE instead of axial RoPE (ablation)Pack3D · Allen 6 structures, 262k/depth1final Dice: 0.7041 -> 0.7151Mixed
use additive 3D relative bias or ALiBi-style bias instead of axial RoPE (ablation)Pack3D · Allen 6 structures, 65k/depth1best macro Dice: 0.7346 -> 0.7055 / 0.7098Hurt
corrupt compressed summaries (shuffle) at inference (ablation)trained Pack3D · Allen 6 structures, 262k/depth8macro Dice: 0.8531 -> 0.6858Hurt

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy