Research map / Vision-language and multimodal / Human and embodied modeling
DreamCatcher: A Wearer-aware Multi-modal Sleep Event Dataset Based on Earables in Non-restrictive Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases DreamCatcher, a 420 hour.person dual-channel earable audio + IMU sleep event dataset recorded with sleep partners, and benchmarks wearer identification, wearer-aware sound classification, and detection.
Problem
Earable-based sleep event detection ignores interference from roommates or partners sharing the bed, and no public earable sleep dataset with multi-sleeper scenarios existed.
Approach
A multi-sleeper dataset with synchronized dual-channel earbud audio (feedback and feedforward mics) and IMU data, annotated with eight sleep events including non-wearer sounds, enabling wearer-aware event identification, classification, and detection benchmarks.
Key results
- Random Forest reached 0.997 accuracy on wearer event identification with leave-one-user-out evaluation.
- BEATs on feedback audio achieved 90.73% accuracy on wearer-aware sound event classification, though macro-F1 was only 57.47%.
- ATST-SED on feedforward audio reached 44.16% macro-F1 in sound event detection, far above SEDNet at about 0.30-0.31%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a random forest instead of a linear SVM | Wearer event identification · Wearer event identification | accuracy: 0.602 -> 0.997 | Helped |
| Use BEATs rather than the other tested classifiers | Wearer-aware sleep sound event classification · Wearer-aware sleep sound event classification | Macro-F1: 57.47 | Helped |
| Use ATST-SED rather than SEDNet | Wearer-aware sleep sound event detection · Wearer-aware sleep sound event detection | Macro-F1: 14.02 -> 24.73 | Helped |
| Use feedback rather than feedforward microphone audio (ablation) | ATST-SED · Wearer-aware sleep sound event detection | Macro-F1: 24.10 -> 24.73 | Helped |
| Use feedback rather than feedforward microphone audio (ablation) | SEDNet · Wearer-aware sleep sound event detection | Macro-F1: 17.98 -> 18.85 | Helped |
| Fuse dual-channel audio rather than use one channel (ablation) | SEDNet · Wearer-aware sleep sound event detection | Macro-F1: 18.85 -> 14.02 | Hurt |
| Balance the training dataset (ablation) | Wav2Vec2.0 · Wearer-aware sleep sound event classification | accuracy: 75.45 -> 70.70 | Hurt |
| Balance the training dataset (ablation) | Wav2Vec2.0 · Wearer-aware sleep sound event classification | Macro-F1: 48.36 -> 46.04 | Hurt |
| Balance the training dataset (ablation) | Wav2Vec2.0 · Wearer-aware sleep sound event classification | recall of rare events | Helped |
| Use feedback rather than feedforward microphone audio (ablation) | Sleep sound event classification models · Wearer-aware sleep sound event classification | accuracy: SleepTSM: 73.01 vs 72.89; EarVAS: 75.75 vs 76.99; BEATs: 90.73 vs 89.64; Wav2Vec2.0: 75.45 vs 73.29; CLAP: 37.04 vs 37.3 | Mixed |
| Use dual-channel audio rather than a single audio channel (ablation) | EarVAS · Wearer-aware sleep sound event classification | accuracy: 75.75 -> 76.64 | Helped |
| Add IMU data to dual-channel audio (ablation) | EarVAS · Wearer-aware sleep sound event classification | accuracy: 76.64 -> 78.07 | Helped |
Surprising findings
- SEDNet trained on dual-channel audio achieved the lowest macro-F1 among configurations, suggesting multi-channel fusion degraded performance.
- Balancing the training set with augmentation improved recall on rare classes such as cough but lowered overall Wav2Vec2.0 accuracy and macro-F1.
Limitations
- Severe class imbalance; cough has very few samples and several detectors never detect it.
- Annotation relies on audio and IMU without video because of privacy, so labels may be inaccurate.
- Prototype earbuds may not be comfortable for all users, especially those with difficulty falling asleep.
Experimental setup
Datasets: DreamCatcher
Benchmarks: DreamCatcher
Metrics: accuracy AUC F1 precision recall MCC error rate FLOPs parameters
Baselines: SleepTSM EarVAS BEATs Wav2Vec2.0 CLAP SEDNet ATST-SED Random Forest Decision Tree AdaBoost Logistic Regression SVM
Builds on: EarSAVAS SleepTSM DCASE sound event detection
Closest papers by meaning
- Beyond Sensor Data: Foundation Models of Behavioral Data from Wearables Improve Health Predictions (ICML 2025 · similarity 0.84)
- Count What You Want: Exemplar Identification and Few-Shot Counting of Human Actions in the Wild (AAAI 2024 · similarity 0.82)
- SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals (ICML 2024 · similarity 0.82)
- Subject-Relative Micro-Motion and Sleep Dynamics for Near-Infrared Video Sleep Staging (NeurIPS 2026 · similarity 0.81)
- OpenMHC: Accelerating the Science of Wearable Foundation Models (NeurIPS 2026 · similarity 0.81)
- SensorLM: Learning the Language of Wearable Sensors (NeurIPS 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.