AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Visual Dependency Transformers: Dependency Tree Emerges From Reversed Attention

Mingyu Ding, Yikang Shen, Lijie Fan, Zhenfang Chen, Zitian Chen, Ping Luo, Joshua B. Tenenbaum, Chuang Gan

CVPR 2023 · Architectures and efficiency › Attention and transformers · approach: Attention head and block redesign · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Reversed attention (dependency tree induction) in weakly-supervised pretrainingDependencyViT-T vs DeiT-Tiny · Pascal-Part mIoUmIoU (%): 18.9 -> 23.2 with maximum spanningImproved
Reversed attention with self-supervised pretraining (vs iBOT baseline)DependencyViT-T · Pascal-Part mIoUmIoU (%): 25.1 -> 28.7Improved
Reversed attention with self-supervised pretraining (vs iBOT baseline)DependencyViT-T · Car-Parts mIoUmIoU (%): 25.7 -> 27.0Improved
Soft dependency mask added to features for saliencyDependencyViT-T (weakly-sup.) · ECSSD maxF_betamaxF_beta (%): DeiT 49.3 vs DependencyViT 62.0Improved
Dependency-based model vs DeiT on unsupervised saliency (weakly-sup.)DependencyViT-T · DUTS maxF_betamaxF_beta (%): 34.2 -> 53.8Improved
Self-supervised instead of weakly-supervised pretraining for saliencyDependencyViT-T · DUT-OMRON maxF_betamaxF_beta (%): 52.0 (weakly) vs 32.5 (self)Hurt
Dynamic pruning of leaf nodes (DependencyViT-Lite)DependencyViT-T ImageNet classification · ImageNet-1K Top-1Top-1 (%): 75.4 -> 73.7 with 0.8 vs 1.3 GFLOPsHurt
Reversed attention (full DependencyViT) vs standard ViTTiny-scale ImageNet classification · ImageNet-1K Top-1Top-1 (%): 72.2 -> 75.4Improved
Pruning fewer tokens (keep 128 vs 64) (ablation)DependencyViT-Lite-T · ImageNet-1K Top-1Top-1 (%): 73.7 (64 tokens) -> 74.9 (128 tokens)Improved
Head selector and message controller (P, M) added to reversed attention (ablation)DependencyViT-T · ablation on tiny modelsaccuracy/ablation score: not reported numerically in provided textImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy