Research map / Architectures and efficiency / RNNs and normalization
Deep Tree Tensor Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Deep Tree Tensor Network (DTTN), a stack of multiplicative antisymmetric interaction modules that captures 2^L-order feature interactions without activations, unfolds into a tree tensor network, and is evaluated on ImageNet-1K and other vision, recsys, and PDE benchmarks.
Problem
Tensor networks have been limited to small image tasks (MNIST, Fashion-MNIST) and are mostly used to compress existing networks rather than capture exponential-order feature interactions at scale.
Approach
Stack antisymmetric interaction modules that use a Hadamard product of two linear/depthwise-conv branches, with no activation functions, so each block is a binary tree node and the whole network is a tree tensor network with parameter sharing and high bond dimension; the paper also shows equivalence to polynomial and quantum-inspired TN models under specific conditions.
Key results
- DTTN-S reaches 77.2% Top-1 on ImageNet-1K at 300 epochs, versus 65.2% for Pi-Nets at 90 epochs (DTTN-S 71.8% at 90 epochs).
- DTTN^dagger-L reaches 82.4% Top-1 on ImageNet-1K with 35.9M params, above DeiT-S/16 (81.2%) and Vim-S (80.5%).
- DTTN-S reaches 0.9930 on MNIST and 0.9236 on Fashion-MNIST, above aResMPS (0.9907 and 0.9142).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DTTN-S instead of Π-Nets | ImageNet-1K, 90 epochs · ImageNet-1K (90 epochs) | Top-1 accuracy (%): 65.2 -> 71.8 | Helped |
| Use DTTN-S instead of Hybrid Π-Nets | ImageNet-1K, 300 epochs · ImageNet-1K (300 epochs) | Top-1 accuracy (%): 70.7 -> 77.2 | Helped |
| Use DTTN†-T instead of MONet-T | ImageNet-1K · ImageNet-1K | Top-1 accuracy (%): 77.0 -> 77.9 | Helped |
| Use DTTN†-S instead of MONet-T | CIFAR-10 · CIFAR-10 | accuracy: 94.8 -> 95.0 | Helped |
| Use DTTN†-S instead of 𝒟-PolyNets | Tiny ImageNet · Tiny ImageNet | accuracy: 61.8 -> 63.8 | Helped |
| Use DTTN†-S instead of MONet-T | ImageNet-100 · ImageNet-100 | accuracy: 87.2 -> 87.7 | Helped |
| Use DTTN-S instead of aResMPS | MNIST and Fashion-MNIST · MNIST | accuracy: 0.9907 -> 0.9930 | Helped |
| Use DTTN-S instead of aResMPS | MNIST and Fashion-MNIST · Fashion-MNIST | accuracy: 0.9142 -> 0.9236 | Helped |
| Add AIM to DeepFM | DeepFM · Criteo | AUC: 80.12 -> 80.44 | Helped |
| Add AIM to FiBiNet | FiBiNet · Criteo | AUC: 80.42 -> 80.97 | Helped |
| Add AIM to DCN-V2 | DCN-V2 · Avazu | AUC: 76.14 -> 76.52 | Helped |
| Increase DTTN depth from L=8 to L=32 at d=256 (ablation) | DTTN · ImageNet-100 | Top-1 (%): 79.2 -> 87.2 | Helped |
Surprising findings
- Removing layer normalization from the AIM is not what limits accuracy; without LN the unfoldable DTTN-S still reaches 77.2% on ImageNet-1K, close to the LN variant.
- Adding LN inside AIM breaks the polynomial unfolding property yet gives gains of 0.4 to 1.8 points, which the authors attribute to LN's second-order statistics.
Limitations
- Computational resource limits prevented pretraining on larger datasets such as JFT-300M.
- Robustness of the model was not evaluated.
- Theoretical analysis is limited; the TN equivalence relies on specific local mappings and the structural differences are judged negligible.
Experimental setup
Datasets: CIFAR-10 Tiny ImageNet ImageNet-100 ImageNet-1K MNIST Fashion-MNIST ADE20K Criteo Avazu
Benchmarks: ImageNet-1K CIFAR-10 Tiny ImageNet ImageNet-100 MNIST Fashion-MNIST Criteo Avazu ADE20K
Metrics: Top-1 accuracy AUC mIoU L2 error latency throughput
Baselines: Pi-Nets Hybrid Pi-Nets PDC R-PolyNets D-PolyNets MONet MPS LoTeNet aResMPS ResNet-50 DeiT-S/16 Swin-T/16
Builds on: Polynomial Networks (Pi-Nets) MONet R-PolyNets MLP-Mixer Tree Tensor Network
Closest papers by meaning
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.86)
- SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective (CVPR 2024 · similarity 0.86)
- tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs) (ICML 2024 · similarity 0.85)
- Position: Tensor Networks are a Valuable Asset for Green AI (ICML 2024 · similarity 0.84)
- SHAP Meets Tensor Networks: Provably Tractable Explanations with Parallelism (NeurIPS 2025 · similarity 0.84)
- Equivariant Matrix Function Neural Networks (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.