Research map / Vision-language and multimodal / Cross-modal learning
Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OPERA, an open respiratory acoustic foundation model pretraining and benchmarking system: curates ~404 hours of respiratory audio, pretrains three SSL encoders (contrastive transformer, contrastive CNN, generative transformer), and benchmarks them on 19 health tasks against general-audio models.
Problem
Respiratory audio foundation models are under-explored; existing acoustic models are pretrained on general audio with very little respiratory content, the one respiratory model (HeAR) is not open, and no standardized benchmark exists for respiratory audio health tasks.
Approach
Pretrain open respiratory-specific acoustic encoders on a curated multi-source respiratory audio corpus using contrastive (transformer and CNN) and masked generative (transformer) SSL, and evaluate all models with a unified linear-probe pipeline over 19 tasks built from 10 public or controlled-access datasets.
Key results
- Best OPERA model (OPERA-CT) reaches MRR 0.5632 overall vs 0.3435 for the best baseline (CLAP), outperforming baselines on 16 of 19 tasks.
- OPERA-CT achieves AUROC above 0.7 on 6 of 12 health condition tasks, versus 3 for CLAP.
- Contrastive models (OPERA-CT/CE) lead on classification while generative OPERA-GT leads on lung function regression (MRR 0.6548 on that group).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on respiratory audio with contrastive learning using a transformer encoder | OPERA-CT · Health condition inference tasks | MRR: 0.4319 -> 0.6944 | Helped |
| Pretrain on respiratory audio with generative learning using a transformer encoder | OPERA-GT · Lung function estimation tasks | MRR: 0.1918 -> 0.6548 | Helped |
| Use contrastive rather than generative pretraining | OPERA respiratory encoders · Health condition inference tasks | MRR: 0.6944 (OPERA-CT) vs 0.4569 (OPERA-GT) | Mixed |
| Use generative rather than contrastive pretraining | OPERA respiratory encoders · Lung function estimation tasks | MRR: 0.6548 (OPERA-GT) vs 0.3381 (OPERA-CT) | Helped |
| Replace the transformer encoder with a lightweight CNN encoder | OPERA-CE vs OPERA-CT · All 19 tasks | MRR: 0.5632 -> 0.4412 | Hurt |
| Fine-tune instead of linear probing with 1749 labeled samples | OPERA models on T4 symptom cough · T4 symptom cough | AUROC: 0.680 -> 0.710 (OPERA-CT); 0.673 -> 0.715 (OPERA-GT) | Helped |
| Fine-tune instead of linear probing with 6648 labeled samples | OPERA-GT on T4 symptom cough · T4 symptom cough | AUROC: 0.673 -> 0.735 | Helped |
| Fine-tune instead of linear probing | OPERA-CT on T12 body position · T12 body position | AUROC: 0.781 -> 0.994 | Helped |
| Combine the contrastive and generative pretraining objectives (ablation) | Hybrid OPERA model · Respiratory benchmark tasks | performance | No clear effect |
Surprising findings
- A simple hybrid of contrastive and generative objectives gave no consistent improvement over either objective alone.
- The lightweight CNN OPERA-CE (4M parameters) underperforms the larger transformer models, suggesting scale matters at this data size.
Limitations
- Some labels (e.g., self-reported COVID-19 status in Task 5) are less precise than clinically validated labels.
- OPERA is not intended for clinical use and has not been validated for safe deployment.
- Pretraining corpus is only ~404 hours, limiting exploration of scaling laws.
Experimental setup
Datasets: COVID-19 Sounds UK COVID-19 COUGHVID ICBHI HF Lung Coswara KAUH Respiratory@TR SSBPR MMLung NoseMic
Benchmarks: OPERA benchmark (19 respiratory health tasks)
Metrics: AUROC MAE MRR
Baselines: OpenSmile (Emobase) VGGish AudioMAE CLAP
Builds on: AudioMAE CLAP Masked autoencoders Hierarchical token-semantic audio transformer EfficientNet
Closest papers by meaning
- Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis (ICLR 2026 · similarity 0.86)
- NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics (ICLR 2025 · similarity 0.85)
- CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models (ICML 2025 · similarity 0.84)
- Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark (CVPR 2024 · similarity 0.83)
- BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics (ICLR 2025 · similarity 0.82)
- VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.