Research map / Architectures and efficiency / RNNs and normalization
SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SILENCE, a lightweight segment-masking encoder with a differentiable mask generator that hides short-term speech details to protect privacy in offloaded speech understanding on microcontrollers.
Problem
Cloud speech understanding leaks private transcript content, but disentanglement-based privacy encoders are too memory- and compute-heavy for resource-constrained wimpy devices.
Approach
Exploits asymmetric dependency: intent classification relies on long-term utterance context while ASR relies on short-term peaky phoneme frames, so masking a learned subset of short segments hides entities while preserving intent; a differentiable L0-style mask generator decides how many and which segments to mask.
Key results
- SILENCE reaches about 81% intent accuracy on SLURP, within ~7% of unprotected AllOffload/OnDevice and 7.2% above PPSLU.
- Under Whisper attack SLU-WER rises from 14.7% to 78.6%; under Azure to 81.6%; white-box Whisper still yields 67.3% WER.
- SILENCE-S uses 394.9KB memory and 912.2ms on STM32H7; on RPI-4B it is 18.1x faster and 134.1x lower memory than PPSLU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SILENCE masking instead of the PPSLU encoder | privacy-preserving SLU · SLURP | intent classification accuracy: +7.2% | Helped |
| mask audio with SILENCE | SLURP intent classification · SLURP | accuracy: less than 7% | Hurt |
| mask audio with SILENCE | SLU audio under Azure ASR · SLURP | WER-SLU: 14.7% -> 81.6% | Helped |
| mask audio with SILENCE | SLU audio under Whisper ASR · SLURP | WER-SLU: 14.7% -> 78.6% | Helped |
| use the SILENCE mask with a white-box attacker | Whisper · SLURP | WER-SLU: 67.3 | Helped |
| use SILENCE-S instead of PPSLU | RPI-4B speech encoding · 4-second audio | encoding latency: 18.1× faster | Helped |
| use SILENCE-S instead of PPSLU | RPI-4B speech encoding · 4-second audio | memory footprint: 134.1× less | Helped |
| use a larger mask generator (SILENCE-L rather than SILENCE-S) (ablation) | SILENCE · SLURP | intent accuracy at the same privacy level | Helped |
| use a larger mask generator (SILENCE-L rather than SILENCE-S) (ablation) | SILENCE · speech encoding | encoding efficiency | Hurt |
| increase the masking threshold (ablation) | SILENCE mask generator · SLURP | privacy protection | Helped |
| increase the masking threshold (ablation) | SILENCE mask generator · SLURP | SLU utility | Mixed |
| use random masking rather than a learnable mask generator (ablation) | SILENCE · SLURP | privacy-utility performance | Mixed |
Surprising findings
- Masking short-term frames destroys entity recoverability even for a white-box attacker fine-tuned on the masking pattern.
- Over 50% of audios sent to Azure were returned as NoMatch, suggesting the masked audio is often unrecognizable.
Limitations
- Large design space for mask generator structures; only three simple variants explored, inheriting the SLU model structure.
- Requires fine-tuning the cloud SLU model for each mask generator.
- Does not defend against short-dependent private keyword or semantic detection attacks.
Experimental setup
Datasets: SLURP FSC LibriSpeech
Benchmarks: SLURP FSC LibriSpeech
Metrics: intent accuracy word error rate entity error rate memory footprint end-to-end latency
Baselines: OnDevice AllOffload VAE PPSLU Random mask
Builds on: HuBERT VAE disentanglement PPSLU SpeechBrain
Closest papers by meaning
- Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation (CVPR 2025 · similarity 0.83)
- DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models (AAAI 2025 · similarity 0.82)
- SilentStriker: Toward Stealthy Bit-Flip Attacks on Large Language Models (NeurIPS 2025 · similarity 0.82)
- ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio–Language Models (NeurIPS 2025 · similarity 0.82)
- ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression (AAAI 2025 · similarity 0.81)
- Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.