Research map / Label-efficient and robust learning / Self-supervised and contrastive

Contrastive Learning from Synthetic Audio Doppelgängers

ICLR 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Inductive bias and minimal representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains audio contrastive encoders (ResNet18) on randomly sampled and perturbed synthesizer parameters ("audio doppelgängers") instead of real recordings, matching or beating VGGSound-trained models on 8 HEAR/ARCH tasks.

Problem

Learning robust general audio representations usually requires large real-world recording datasets, and standard audio augmentations only approximate real variation in sounds.

Approach

Generate positive pairs by sampling random synthesizer parameter vectors and adding Gaussian perturbations scaled by a single factor delta, so positives differ causally in timbre, pitch, and envelopes; train with alignment and uniformity losses on the fly without storing data.

Key results

• Best synthetic model (Voice, delta=0.25) reaches 58.90 on ESC-50 and 66.71 on UrbanSound8K, outperforming VGGSound SSL with augmentation on these tasks.
• Voice delta=0.25 is the top synthetic-trained model on 5/8 tasks and beats temporal-jitter VGGSound training on 6/8 tasks.
• Synthetic sounds are more causally uncertain than VGGSound clips, and Voice achieves the lowest FAD vs. target data on 5/6 tasks, with VGGSound closer on ESC-50.

Measured effects reported in the paper

Surprising findings

• Synthetic-only pretraining with no real audio outperforms real VGGSound pretraining with augmentations on several tasks.
• Adding augmentations on top of audio doppelgänger training gives no significant benefit, while it helps greatly for delta=0 synthetic training.

Limitations

• Evaluated only with ResNet18 encoders; larger encoders such as AST left for future work.
• Only a single isotropic Gaussian perturbation scheme was studied, with no anisotropic or learned perturbations.
• Evaluation limited to classification benchmarks; representation disentanglement not assessed.

Experimental setup

Datasets: VGGSound synthetic audio from SynthAX Voice, VoiceFM, ParametricSynth synthesizers ESC-50 UrbanSound8K VIVAE NSynth CREMA-D FSD50k Vocal Imitations LibriCount

Benchmarks: ESC-50 UrbanSound8K VIVAE NSynth Pitch 5h CREMA-D FSD50k Vocal Imitations LibriCount

Metrics: accuracy FAD cosine similarity alignment and uniformity losses

Baselines: VGGSound SSL with augmentations VGGSound SSL with temporal jitter Random initialization MS-CLAP GURA VGGSound supervised ResNet18 HEAR leaderboard top models

Builds on: SynthAX alignment and uniformity objective SpecAugment VGGish

Closest papers by meaning

• Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.87)
• Creative Text-to-Audio Generation via Synthesizer Programming (ICML 2024 · similarity 0.85)
• Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.85)
• CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR 2024 · similarity 0.84)
• ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.84)
• Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.84)