Research map / Architectures and efficiency / RNNs and normalization
Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper
Problem
Understanding the function space and inductive bias of multi-output (vector-valued) neural networks trained with weight decay, and how many neurons are needed in shallow and deep networks.
Approach
Defines a vector-valued variation norm whose representational cost is the sum of L2 norms of output weights; this equals weight-decay regularization for homogeneous activations via the neural balance theorem, which yields neuron sharing across outputs, data-dependent width bounds, and a convex multi-task lasso compression procedure.
Key results
- Compressed VGG-19 last layer from 512 to 99 neurons with train loss 0.0002 and test accuracy 93.90% (original 93.91%).
- Compressed AlexNet last layer from 4096 to 555 neurons, with test accuracy 72.14% vs 72.17% original.
- Weight-decay-trained neuron sharing experiment: 5 active neurons under weight decay vs 85 under l1 and 130 with no regularization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress the last layer | VGG-19 · CIFAR-10, last layer | width: 512 -> 99 | Helped |
| Compress the last layer | VGG-19 · CIFAR-10, last layer | test accuracy: 93.91% -> 93.90% | No clear effect |
| Compress the last layer | AlexNet · CIFAR-10, last layer | width: 4096 -> 555 | Helped |
| Compress the last layer | AlexNet · CIFAR-10, last layer | test accuracy: 72.17% -> 72.14% | No clear effect |
| Compress the penultimate layer | AlexNet · CIFAR-10, penultimate layer | width: 4096 -> 2369 | Helped |
| Compress the penultimate layer | AlexNet · CIFAR-10, penultimate layer | test accuracy: 72.17% -> 72.16% | No clear effect |
| Compress the first FC layer | AlexNet · CIFAR-10, first FC layer | width: 9216 -> 8208 | Helped |
| Compress the first FC layer | AlexNet · CIFAR-10, first FC layer | test accuracy: 72.17% -> 72.18% | No clear effect |
| Compress all three FC layers | AlexNet · CIFAR-10, last three layers | test accuracy: 72.17% -> 72.13% | No clear effect |
| Train with weight decay rather than ℓ1 regularization or no regularization | Three-output shallow ReLU network · Synthetic two-dimensional dataset | active neurons: 85 -> 5; 130 -> 5 | Helped |
Limitations
- Theory is developed for homogeneous activations (e.g., ReLU) and the regularized-bias setting; the unregularized bias case is only discussed as an extension.
- Experiments are small-scale (synthetic data, CIFAR-10 with VGG-19 and AlexNet), and the sparsity bound is not always attained in experiments.
Experimental setup
Datasets: CIFAR-10 synthetic two-dimensional dataset
Benchmarks: CIFAR-10
Metrics: training loss test accuracy sum of squared weights number of active neurons layer width
Baselines: l1 regularization no regularization
Builds on: neural balance theorem scalar-valued variation spaces Maurey-Jones-Barron lemma multi-task lasso
Closest papers by meaning
- Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020 · similarity 0.84)
- Compressing Neural Networks using the Variational Information Bottleneck (ICML 2018 · similarity 0.84)
- Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (NeurIPS 2021 · similarity 0.83)
- A New Neural Kernel Regime: The Inductive Bias of Multi-Task Learning (NeurIPS 2024 · similarity 0.82)
- Auto-Compressing Networks (NeurIPS 2025 · similarity 0.81)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.