Research map / Architectures and efficiency / RNNs and normalization

Harnessing the Power of SVD: An SVA Module for Enhanced Signal Classification

AAAI 2024 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SVA, an SVD-based attention module that reweights singular values of intermediate features, plus clustering and an orthogonality loss to stabilize training, and reports gains for ResNet and ViT-Lite on signal classification.

Problem

Signal classification (modulation recognition, specific emitter identification, EEG) suffers from noisy, degraded signals, and existing attention modules enhance features explicitly without exploiting the data's internal subspace structure; embedding SVD into deep networks also causes gradient explo

Approach

Insert an SVD decomposition layer into a network and learn an input-dependent attention over singular values (Λ = σ(f3(ReLU(f2(f1(vec(X)))))) ), reweighting the semantic subspaces; reduce SVD cost and gradient explosion by clustering feature maps into k centers and adding an orthogonality loss on YY^T.

Key results

• On RML2016.10a, ResNet18 with SVA reaches 61.44% OA versus 58.84% for the base ResNet18 while reducing parameters from 3.849M to 3.775M and MFlops from 87.41 to 83.92.
• On HisarMod2019, ResNet34 with SVA reaches 84.59% OA versus 74.04% for base ResNet34.
• Adding clustering and orthogonal loss to SVA reduces training time per sample by 71.8% relative to SVA without clustering, while raising accuracy to 61.49% from 58.83% (ablation on RML2016.10a with ResNet18 per Table 4).

Measured effects reported in the paper

Surprising findings

• Plain SVD inserted into the network frequently fails due to gradient explosion and ill-conditioning; the SVA design with clustering and orthogonal loss substantially reduces these failures.
• Inserting SVA after layer2 of ResNet gives the largest gains, while insertion after conv1 or later layers gives smaller or negative change.

Limitations

• Reported performance uses a fixed cluster count k=10 that is not always optimal (best k=6 in Table 3), indicating sensitivity to hyperparameters.
• SVD cost grows as O(n^3) and the method relies on clustering to reduce matrix size, which limits scalability to large feature dimensions.
• Some SVA results are below the best attention baseline (e.g., LSRWR with ResNet18: SVA 73.62 vs ECA 73.24, SENet 73.04 on ResNet18 but SVA lower than SENet on ResNet34 74.93 vs 74.26 is higher; mixed gains) and the paper does not provide code release.

Experimental setup

Datasets: RML2016.10a HisarMod2019 LSRWR sleep-edf-2002 sleep-edf-2013 HAR

Benchmarks: RML2016.10a HisarMod2019 LSRWR sleep-edf-2002 sleep-edf-2013 HAR

Metrics: overall accuracy parameter count MFlops training/testing time

Baselines: SENet CBAM ECA SimAM MCLDNN PETCGDNN AttnSleep DeepSleepNet

Builds on: SENet CBAM ECA SimAM

Closest papers by meaning

• A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks (ICML 2024 · similarity 0.80)
• Attention-based PCA (NeurIPS 2026 · similarity 0.79)
• Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models (CVPR 2025 · similarity 0.79)
• Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.79)
• Deep Principal Support Vector Machines for Nonlinear Sufficient Dimension Reduction (ICML 2025 · similarity 0.78)
• Transformers are almost optimal metalearners for linear classification (NeurIPS 2025 · similarity 0.78)