Research map / Label-efficient and robust learning / Recognition and long-tail
Towards A Translative Model of Sperm Whale Vocalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes the music-pretrained VampNet masked acoustic token model on animal and sperm whale coda audio to generate and acoustically translate sperm whale codas, evaluated by FAD, expert listening tests, and downstream classification.
Problem
Modeling and generating sperm whale codas, and translating audio from other domains into coda-like acoustic style, with generative models that also learn useful representations for classification; prior bioacoustic generators were GANs that cannot condition on an audio prompt.
Approach
Adapts a pretrained masked acoustic token transformer (VampNet) to sperm whale codas in two LoRA finetuning phases (domain adaptation on animal sounds, then species-specific finetuning on coda datasets), so that masked-token iterative decoding gives audio-prompted translation and generation, with its embeddings reused for classification.
Key results
- Expert audio-only 2AFC accuracy averaged 81% (kappa=0.41), rising to 83% with spectrograms, indicating synthetic codas are hard but not perfectly indistinguishable.
- WhAM embeddings reach 87.4% on rhythm and 85.2% on vowel classification, outperforming random and majority baselines though below AVES and BirdNET on most tasks.
- After translation through WhAM, outputs from four non-whale species and digital beeps become FAD-indistinguishable (below 0.21) from natural codas.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WhAM embeddings instead of random AVES embeddings | two-layer classifier · Coda detection | accuracy: 60.9 -> 91.3±0.2 | Helped |
| Use WhAM embeddings instead of random AVES embeddings | two-layer classifier · Rhythm type | accuracy: 66.3 -> 87.4±1.6 | Helped |
| Use WhAM embeddings instead of random AVES embeddings | two-layer classifier · Social Unit | accuracy: 42.5 -> 70.5±5.6 | Helped |
| Use WhAM embeddings instead of random AVES embeddings | two-layer classifier · Vowel type | accuracy: 66.3 -> 85.2±2.5 | Helped |
| Use WhAM rather than AVES embeddings | two-layer classifier · Coda detection | accuracy: 92.8±0.1 -> 91.3±0.2 | Hurt |
| Use WhAM rather than AVES embeddings | two-layer classifier · Rhythm type | accuracy: 90.4±1.6 -> 87.4±1.6 | Hurt |
| Use WhAM rather than AVES embeddings | two-layer classifier · Social Unit | accuracy: 92.0±0.7 -> 70.5±5.6 | Hurt |
| Use WhAM rather than AVES embeddings | two-layer classifier · Vowel type | accuracy: 91.8±2.9 -> 85.2±2.5 | Hurt |
| Show spectrograms in addition to audio | expert discrimination of WhAM codas · Expert 2AFC | accuracy: 81% -> 83% | Helped |
| Translate walrus vocalizations into coda-like audio | WhAM · Walrus-to-coda mixed classification | correctly flagged synthetic audio: 75% | Mixed |
| Fine-tune VampNet for sperm whale codas (ablation) | WhAM embeddings · Downstream classification tasks | classification utility | No clear effect |
| Add species-specific fine-tuning on sperm whale codas (ablation) | WhAM · Audio translation to sperm whale vocalization acoustics | translation capability | Helped |
Surprising findings
- Experts flagged walrus-to-coda translations as synthetic only 75% of the time, less than digital beeps or coda-to-coda outputs.
- Species-specific finetuning was essential for translation, while domain-adaptation-stage finetuning did not significantly change downstream embedding utility compared to base VampNet.
Limitations
- Only the masked acoustic token model is finetuned while the codec stays fixed, possibly missing vowel-like spectral features in the 3.7-5.7 kHz band.
- Expert perceptual study used only five experts; synthetic clicks showed unnatural onset/decay and echolocation-like timing and background noise artifacts.
- Coda dataset is small (about 10k codas) and the data cannot be fully released due to consent constraints.
Experimental setup
Datasets: Freesound Dataset (animal subset) AudioSet (animal subset) Watkins Marine Mammal Sound Database BirdSet Dominica Sperm Whale Project (DSWP) CETI sperm whale coda dataset
Benchmarks: Fréchet Audio Distance on coda vs. translated audio expert perceptual study (2AFC and mixed classification) coda detection rhythm type classification social unit classification vowel type classification
Metrics: Fréchet Audio Distance accuracy Fleiss's kappa
Baselines: AVES BirdNET CLAP Random embedding baseline Majority-class baseline
Builds on: VampNet Descript Audio Codec LoRA iterative parallel decoding
Closest papers by meaning
- OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations (NeurIPS 2026 · similarity 0.87)
- NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics (ICLR 2025 · similarity 0.87)
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.84)
- Contrastive Learning from Synthetic Audio Doppelgängers (ICLR 2025 · similarity 0.83)
- Vocal Call Locator Benchmark (VCL) for localizing rodent vocalizations from multi-channel audio (NeurIPS 2024 · similarity 0.83)
- NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.