Research map / Large language models / Language, safety and interpretability
NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NatureLM-audio, a BEATs + Q-Former + LoRA-tuned Llama-3.1-8B audio-language model trained on bioacoustic, speech and music data, plus the BEANS-Zero benchmark, reporting zero-shot SOTA on several bioacoustic tasks.
Problem
Audio-language models have been applied to speech, music and general audio but not to bioacoustics, where annotated data is scarce and tasks such as species classification, detection, call-type, lifestage and individual counting need generalization to unseen taxa.
Approach
Train a single audio-to-text LLM on a curated mix of bioacoustic, general audio, speech and music text-audio pairs in a two-stage curriculum (focal species classification first, then broader tasks), showing that speech and music data transfer to bioacoustic tasks and generalize to unseen species, genera and families.
Key results
- On BEANS-Zero zero-shot, NatureLM-audio reaches 0.788 accuracy on watkins and 0.778 on cbi, state of the art on 7 of 9 classification/detection datasets.
- On unseen species, unseen genus and unseen family classification with taxonomic names, accuracy is 0.343, 0.148 and 0.308 respectively.
- Captioning SPIDEr of 0.532 versus 0.009 for SALMONN; zebra finch individual counting 0.655 versus 0.604 SotA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Predict taxonomic names instead of common or scientific names | NatureLM-audio · unseen-species, unseen-genus, and unseen-family | accuracy: 0.181 -> 0.343; 0.116 -> 0.148; 0.035 -> 0.308 | Helped |
| Fine-tune BEATs instead of freezing it (ablation) | NatureLM-audio · watkins | accuracy: 0.490 -> 0.723 | Helped |
| Fine-tune BEATs instead of freezing it (ablation) | NatureLM-audio · cbi | accuracy: 0.401 -> 0.680 | Helped |
| Fine-tune BEATs instead of freezing it (ablation) | NatureLM-audio · unseen-species | accuracy: 0.184 -> 0.320 | Helped |
| Fine-tune BEATs instead of freezing it (ablation) | NatureLM-audio · unseen-genus | accuracy: 0.073 -> 0.124 | Helped |
| Fine-tune BEATs instead of freezing it (ablation) | NatureLM-audio · unseen-family | accuracy: 0.186 -> 0.306 | Helped |
| Add speech and music tasks during training (ablation) | NatureLM-audio · zf-indv | accuracy: 0.500 -> 0.677 | Helped |
| Add speech and music tasks during training (ablation) | NatureLM-audio · unseen-family | accuracy: 0.330 -> 0.239 | Hurt |
| Add speech and music tasks during training (ablation) | NatureLM-audio · unseen-species | accuracy: 0.354 -> 0.322 | Hurt |
| Add speech and music tasks during training (ablation) | NatureLM-audio · cbi | accuracy: 0.750 -> 0.748 | No clear effect |
| Add recording metadata and recordist notes | NatureLM-audio · cbi | accuracy: 0.776 -> 0.798 | Helped |
| Add speech and music tasks during training (ablation) | NatureLM-audio · BEANS-Zero tasks overall | accuracy | Mixed |
Surprising findings
- Removing speech and music training data gave random performance (50.0%) on zebra finch individual counting, while the speech-trained model reached 67.7%, supporting cross-domain transfer.
- On the gibbons detection task NatureLM-audio scored 0.005 F1, below BioLingual's 0.018, and on hiceas it was second to BioLingual.
Limitations
- Bird data is overrepresented, creating bias toward bird vocalizations.
- Possible misuse for poaching and unknown ecological impacts of passive monitoring.
- Relies on Levenshtein distance mapping of free-text outputs to labels, which may be suboptimal for general audio.
Experimental setup
Datasets: WavCaps AudioCaps Clotho NSynth LibriTTS VCTK Xeno-canto iNaturalist Animal Sound Archive Watkins Sapsucker Woods Barkley Canyon
Benchmarks: BEANS-Zero BirdSet ESC50 BEANS
Metrics: accuracy macro F1 SPIDEr
Baselines: BioLingual SALMONN Qwen2-audio CLAP-LAION BirdNET Perch LLM without audio
Builds on: SALMONN BEATs Llama 3.1 BEANS BioCLIP
Closest papers by meaning
- OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations (NeurIPS 2026 · similarity 0.89)
- BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics (ICLR 2025 · similarity 0.89)
- The iNaturalist Sounds Dataset (NeurIPS 2024 · similarity 0.89)
- Vocal Call Locator Benchmark (VCL) for localizing rodent vocalizations from multi-channel audio (NeurIPS 2024 · similarity 0.88)
- Towards A Translative Model of Sperm Whale Vocalization (NeurIPS 2025 · similarity 0.87)
- Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.