Research map / Architectures and efficiency / Attention and transformers
Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Direct3D-S2 proposes Spatial Sparse Attention (SSA) and a fully sparse SDF VAE to train a sparse-voxel image-to-3D diffusion transformer at 1024^3 resolution on 8 GPUs.
Problem
High-resolution 3D shape generation with volumetric (sparse SDF) latents is limited by the quadratic cost of full attention over very long token sets (over 100k tokens at 1024^3) and by heterogeneous VAE representations that hurt training efficiency.
Approach
Adapt Native Sparse Attention to 3D sparse voxels by partitioning tokens into spatial subgrid blocks (rather than 1D index blocks), combining sparse 3D compression, top-k spatial blockwise selection, and sparse 3D window attention with learned gates; pair it with a symmetric sparse-voxel SDF VAE and a foreground-only sparse image conditioning.</1>
Key results
- SSA achieves 3.9x forward and 9.6x backward speedup over FlashAttention-2 at 128k tokens.
- Direct3D-S2 reaches ULIP-2 0.3111 vs. 0.2825 for Trellis on the image-to-3D comparison, the best of all compared methods on all three metrics.
- Training at 1024^3 resolution is possible on 8 A100 GPUs, versus at least 32 GPUs for prior methods at 256^3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Direct3D-S2 instead of Trellis | Image-to-3D generation · Image-to-3D benchmark | ULIP-2: 0.2825 -> 0.3111 | Helped |
| Use Direct3D-S2 instead of TripoSG | Image-to-3D generation · Image-to-3D benchmark | Uni3D: 0.3870 -> 0.3931 | Helped |
| Use Direct3D-S2 instead of Trellis | Image-to-3D generation · Image-to-3D benchmark | OpenShape: 0.1732 -> 0.1752 | Helped |
| Add sparse 3D compression to the sparse 3D window module (ablation) | Direct3D-S2 SSA at 512³ · Image-to-3D generation | mesh quality | No clear effect |
| Add spatial blockwise selection to sparse 3D window and compression (ablation) | Direct3D-S2 SSA at 512³ · Image-to-3D generation | mesh quality | Helped |
| Remove the sparse 3D window module (ablation) | Direct3D-S2 SSA at 512³ · Image-to-3D generation | convergence speed | Hurt |
| Increase generation resolution from 256³/384³ to 512³ (ablation) | Direct3D-S2 · Image-to-3D generation | mesh quality | Helped |
| Increase generation resolution from 512³ to 1024³ (ablation) | Direct3D-S2 · Image-to-3D generation | image alignment | Helped |
| Use 1D fixed-length block partitioning that ignores spatial coherence instead of SSA (ablation) | Direct3D-S2 DiT at 512³ · Image-to-3D generation | surface smoothness | Hurt |
| Use packed full attention instead of SSA (ablation) | Direct3D-S2 DiT at 512³ · Image-to-3D generation | surface artifacts | Hurt |
| Exclude non-foreground image conditioning tokens (ablation) | Direct3D-S2 at 512³ · Image-to-3D generation | image alignment | Helped |
Surprising findings
- Adding the compression module alone changed little; its main role is producing block scores for selection.
- Omitting the window module did not hurt final quality much but slowed convergence.
Limitations
- Forward-pass speedup is notably smaller than backward speedup due to top-k sorting overhead.
- Evaluation relies on alignment metrics (ULIP-2, Uni3D, OpenShape) and a 40-participant user study rather than standard geometric benchmarks.
Experimental setup
Datasets: Objaverse Objaverse-XL ShapeNet
Benchmarks: Direct3D-S2 image-to-3D challenge set (professional community images)
Metrics: ULIP-2 score Uni3D score OpenShape score user study score (1-5) forward/backward runtime
Baselines: Trellis Hunyuan3D 2.0 TripoSG Hi3DGen FlashAttention-2 Dora
Builds on: Native Sparse Attention Trellis rectified flow FlashAttention-2 3DShape2Vecset
Closest papers by meaning
- Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling (NeurIPS 2025 · similarity 0.90)
- MeshFormer : High-Quality Mesh Generation with 3D-Guided Reconstruction Model (NeurIPS 2024 · similarity 0.89)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.88)
- Scaling Diffusion Mamba with Bidirectional SSMs for Efficient 3D Shape Generation (AAAI 2025 · similarity 0.88)
- PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.