Research map / Vision-language and multimodal / Human and embodied modeling

Learning 1D Causal Visual Representation with De-focus Attention Networks

NeurIPS 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes De-focus Attention Networks that add learnable bandpass filters (learnable decay + RoPE), large scheduled drop path, and an auxiliary pooled-feature loss so 1D causal ViT/Mamba/RetNet match 2D non-causal ViTs on ImageNet, COCO detection and CLIP.

Problem

1D causal vision models (causal ViTs, SSMs like Mamba) underperform 2D non-causal vision models because their attention over-concentrates on a few visual tokens, limiting diverse feature extraction and gradient flow, which hampers unified vision-language modeling.

Approach

Attach learnable exponential spatial decay and learnable RoPE to causal attention so they act as learnable bandpass filters producing diverse attention patterns, combined with a linearly scheduled large drop path rate and an auxiliary cross-entropy loss on average-pooled image tokens to spread attention and gradients across tokens.

Key results

• De-focus Mamba-Base reaches 82.0% ImageNet-1K top-1 with causal modeling, matching DeiT-Base non-causal at 81.8%.
• De-focus Mamba-Base with DINO on COCO at 36 epochs achieves 53.2 AP box versus 52.3 for DeiT-Base.
• Ablation: learnable decay plus learnable RoPE raises Mamba-Base accuracy from 75.2 (neither) to 81.2, and linear drop path (0.1 to 0.7) plus average-pooled auxiliary loss yields 82.0.

Measured effects reported in the paper

Surprising findings

• Average-pooled token features alone as the loss perform poorly (77.2%), yet serve as an effective auxiliary loss on top of the CLS loss.
• Data-dependent decay in Mamba adds only about 0.1% over learnable fixed decay in the ablation.

Limitations

• CLIP zero-shot result (62.7%) is slightly below OpenAI CLIP and OpenCLIP baselines at ViT-B/32 scale.
• Some non-causal baselines remain stronger on specific metrics, e.g., DeiT-Base has higher AP50 on COCO.
• Large-model results rely on ImageNet-21K pre-training and resolution transfer tricks.

Experimental setup

Datasets: ImageNet-1K ImageNet-21K MS-COCO Laion-400M

Benchmarks: ImageNet-1K MS-COCO

Metrics: top-1 accuracy AP box AP50 box AP75 box zero-shot top-1 accuracy

Baselines: DeiT Vision Mamba Vision RWKV Mamba-ND S4ND-ViT RetNet ResNet-50 OpenAI CLIP OpenCLIP ViT-Large

Builds on: Mamba RetNet RoPE xPos DeiT

Closest papers by meaning

• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation (AAAI 2026 · similarity 0.88)
• Rethinking Causal Mask Attention for Vision-Language Inference (ICLR 2026 · similarity 0.87)
• See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.86)
• From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.86)
• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.86)
• Same or Not? Enhancing Visual Perception in Vision-Language Models (CVPR 2026 · similarity 0.86)