Research map / Architectures and efficiency / RNNs and normalization

ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADMN, a multimodal network with LayerDrop-trained adaptive-depth backbones and a QoI-aware controller that allocates a strict total layer budget across modalities per sample, matching accuracy of full models at up to 75% fewer FLOPs.

Problem

Multimodal networks waste compute under strictly bounded, time-varying compute budgets and allocate static depth regardless of per-modality input quality (corruption), while dynamic networks optimize only average-case compute.

Approach

Train a single backbone with LayerDrop (MAE pretraining plus multimodal finetuning) so layers can be dropped at test time, then learn a controller that, for a hard total layer budget L, allocates layers across modality backbones per sample using corruption-aware supervision or an autoencoder-based initialization; top-L discretization with Gumbel-Softmax and straight-through estimator makes this tr

Key results

• On GDTM Blur, ADMN at 6 layers localizes at 11.8 cm vs 9.4 cm upper bound while reducing FLOPs by ~75% and latency by ~60%.
• On GDTM Lowlight at 8 layers, ADMN and ADMN_AE localize within 5 cm of the upper bound (upper bound 18.8 cm), while the next best baseline incurs almost 27 cm error.
• On AVE (rain + background noise) at 12 layers, ADMN reaches 67.48% vs 71.19% upper bound; on MM-Fi Gaussian at 16 layers, ADMN_AE reaches 44.40% vs 44.44% upper bound.

Measured effects reported in the paper

Surprising findings

• Removing the controller's corruption supervision and autoencoder initialization (task loss only) substantially degrades accuracy, showing task loss alone fails to learn QoI-aware allocation.
• The controller allocated all resources to the image modality under heavy blur where the car is not visible to humans, because the visual backbone still localizes it.

Limitations

• A separate controller must be trained and loaded for each layer budget; the universal controller is only preliminary.
• Batched inference is not yet supported since per-sample layer allocations vary.
• Evaluated on corruptions that are synthetic and on a small set of datasets with two or three modalities.

Experimental setup

Datasets: GDTM MM-Fi AVE ImageNet-1k AudioSet

Benchmarks: GDTM MM-Fi AVE

Metrics: localization error (cm) classification accuracy GFLOPs latency (ms)

Baselines: Upper Bound Naive Allocation Naive Scratch Modality Network Selection (MNS) Image Only Depth Only Audio Only

Builds on: LayerDrop Masked Autoencoder (MAE) Gumbel-Softmax Vision Transformer Once-For-All

Closest papers by meaning

• Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.83)
• Adaptive Depth Networks with Skippable Sub-Paths (NeurIPS 2024 · similarity 0.83)
• Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.82)
• Understanding Unimodal Bias in Multimodal Deep Linear Networks (ICML 2024 · similarity 0.82)
• Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.82)
• Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.82)