Research map / Architectures and efficiency / Pruning and efficient tuning

LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition

ICLR 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Efficient ViT adaptation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LUM-ViT, a ViT whose patch-embedding is a learnable under-sampling mask applied to DMD-based optical computation with kernel-level binarized weights and a three-stage fine-tuning strategy, enabling pre-acquisition optical modulation of hyperspectral/image data.

Problem

Bandwidth constraints during signal acquisition impede real-time detection, especially for hyperspectral data whose volume is huge; compressive sensing performs poorly at low under-sampling rates and cannot exploit prior knowledge of the target.

Approach

Replace the ViT patch-embedding with a DMD-compatible single-channel binarized convolution whose (patch, kernel) outputs are gated by a learnable binary mask trained with Gumbel-Softmax and a rate loss, so only retained points are optically acquired; weights are binarized per kernel and training is split into three stages.

Key results

• At 10% under-sampling LUM-ViT keeps accuracy loss within 1.8% on ImageNet-1k; at 2% the loss stays under 5.5%.
• Real-world DMD experiment on LUM-ViT-2%: 64.4% accuracy on 7-color e-ink display (74.7% excluding device-induced anomalies) vs. 75.7% for MAE-base under the same 7-color conversion.
• Hyperspectral classification overall accuracy after mask: 86.2% Indian Pines, 88.6% Pavia University, 99.1% Salinas.

Measured effects reported in the paper

Surprising findings

• Full-layer binarization degraded accuracy less than expected only by about 0.6 points; the paper emphasizes kernel-level binarization is needed to keep performance.
• Attempts to train the binary layer and mask concurrently were unsuccessful, requiring a three-stage schedule.

Limitations

• Mask is fixed after training and must be determined before target acquisition; no dynamic masking.
• Requires model adaptation (reconfigured channel weights and fine-tuning) to move from RGB to hyperspectral/7-color hardware.
• Real-world experiment uses only 371 ImageNet test images and a 7-color e-ink display, with a 4-5% hardware-induced drop.

Experimental setup

Datasets: ImageNet-1k Indian Pines Pavia University Salinas

Benchmarks: ImageNet-1k Indian Pines Pavia University Salinas

Metrics: Top-1 accuracy overall accuracy

Baselines: DU-ViT Random-mask-ViT Mag-mask-ViT CS-method (OMP reconstruction + ViT) MAE-base

Builds on: Vision Transformer MAE Gumbel-Softmax DynamicViT AdaViT

Closest papers by meaning

• LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition (AAAI 2024 · similarity 0.83)
• LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift (NeurIPS 2026 · similarity 0.83)
• Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.83)
• ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.82)
• Luminance-Aware Statistical Quantization: Unsupervised Hierarchical Learning for Illumination Enhancement (NeurIPS 2025 · similarity 0.82)
• SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction (CVPR 2024 · similarity 0.82)