Research map / Label-efficient and robust learning / Recognition and long-tail
ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ND-MRM, a spiking neural network for multisensory emotion recognition that splits hidden neurons into unisensory (visual/auditory) and multisensory types and constrains cross-layer connections via masks, reaching 99.63% on RAVDESS and 97.67% on eNTERFACE'05.
Problem
Multisensory recognition lacks efficient cross-sensory interaction in artificial networks; prior brain-inspired models (Convergence, Enhancement, Synch-Graph) either omit cross-sensory interaction or are fragile to noise, while transformer models like MulT are computationally heavy.
Approach
Mimics neuronal diversity in the brain by labeling hidden-layer spiking neurons as visual-unisensory, auditory-unisensory, or multisensory (using spike firing statistics from two unisensory recognizers), and uses connection masks that allow only biologically motivated transmissions so that features cross senses via multisensory neurons.
Key results
- ND-MRM reaches 99.63% weighted accuracy on RAVDESS (8 classes), above Synch-Graph's 98.30% reported on 6 classes.
- ND-MRM reaches 97.67% on eNTERFACE'05, outperforming Synch-Graph (96.82%) by 0.85 points.
- Multisensory-only MRM ablation drops to 94.18% on RAVDESS and 93.33% on eNTERFACE'05; the best neuron-type ratio was rho=0.7 (about 33% multisensory neurons).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unisensory neurons alongside multisensory neurons (ablation) | ND-MRM · RAVDESS | weighted accuracy: 5.45% | Helped |
| Add unisensory neurons alongside multisensory neurons (ablation) | ND-MRM · eNTERFACE’05 | weighted accuracy: 4.34% | Helped |
| Set ρ to 0.7 rather than 0.1 (ablation) | ND-MRM · RAVDESS | weighted accuracy: 0.8989 -> 0.9963 | Helped |
| Set ρ to 0.7 rather than 0.9 (ablation) | ND-MRM · RAVDESS | weighted accuracy: 0.9426 -> 0.9963 | Helped |
| Set ρ to 0.7 rather than 0.1 (ablation) | ND-MRM · eNTERFACE’05 | weighted accuracy: 0.9124 -> 0.9767 | Helped |
| Set ρ to 0.7 rather than 0.9 (ablation) | ND-MRM · eNTERFACE’05 | weighted accuracy: 0.9147 -> 0.9767 | Helped |
| Increase ρ from 0.7 to 0.9 (ablation) | ND-MRM · RAVDESS and eNTERFACE’05 | weighted accuracy: 0.9963 -> 0.9426; 0.9767 -> 0.9147 | Hurt |
Surprising findings
- Increasing the multisensory neuron proportion to about 69% (rho=0.9) lowers accuracy below the rho=0.7 setting, attributed to imbalance between unisensory and multisensory neurons.
Limitations
- Evaluated only on visual and audio modalities for emotion recognition.
- Not an end-to-end model; neuron categorization relies on separately trained unisensory recognizers.
- Performance depends on the hyperparameter rho, with sensitivity shown in ablation.
Experimental setup
Datasets: RAVDESS eNTERFACE'05
Benchmarks: RAVDESS eNTERFACE'05
Metrics: accuracy weighted accuracy
Baselines: Convergence Enhancement Synch-Graph MR-SNN MulT MRM (multisensory-only ablation)
Builds on: leaky integrate-and-fire neuron MR-SNN Convergence and Enhancement models Synch-Graph
Closest papers by meaning
- NeurIPT: Foundation Model for Neural Interfaces (NeurIPS 2025 · similarity 0.82)
- Neuron-based Multifractal Analysis of Neuron Interaction Dynamics in Large Models (ICLR 2025 · similarity 0.82)
- CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.82)
- The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks. (ICLR 2024 · similarity 0.82)
- Modeling Dynamic Neural Activity by combining Naturalistic Video Stimuli and Stimulus-independent Latent Factors (NeurIPS 2025 · similarity 0.81)
- Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.