Research map / Vision-language and multimodal / Human and embodied modeling
Massive Sound Embedding Benchmark (MSEB)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MSEB, an extensible benchmark library with eight audio-embedding super tasks and a new SVQ spoken-query dataset, showing large headroom between sound-based and text-oracle baselines.
Problem
Existing audio evaluation is fragmented across speech and non-speech communities, with no standardized benchmark that measures general-purpose sound embeddings across retrieval, reasoning, classification, clustering, reconstruction and other tasks, including efficiency.
Approach
A model-agnostic, MTEB-style benchmark framework where any encoder (end-to-end, cascade, or multi-tower) produces embeddings evaluated on eight audio task families without task-specific fine-tuning, plus efficiency metrics (compression ratio, FLOPS), and a new 26-locale spoken query dataset (SVQ) supporting multiple tasks from one source.
Key results
- Gemini-based cascade reranking on SVQ reaches 0.86 mAP for en-AU but drops to 0.11 mAP for ml-IN.
- Whisper Large v3 transcription WER ranges from 7.7% (Arabic, clean) to over 100% (Malayalam).
- EnCodec reconstruction FAD on FSD50K reaches about 778k, showing reconstruction is far from solved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Whisper ASR transcript with ground-truth transcript | Gemma 3 reasoning on SVQ · Cross-Language reasoning, ml-IN | F1: 0.12 -> 0.54 | Helped |
| Replace Whisper ASR transcript with ground-truth transcript | Gemma 3 reasoning on SVQ · In-Language reasoning, Bengali | F1: 0.54 -> 0.65 | Helped |
| Replace Whisper ASR transcript with ground-truth transcript | GeminiEmbedding intent classification · Speech-MASSIVE, German | accuracy: 0.83 -> 0.84 | Helped |
| Replace Whisper ASR transcript with ground-truth transcript | GeminiEmbedding intent classification · Speech-MASSIVE, Arabic | accuracy: 0.67 -> 0.77 | Helped |
| Use audio input instead of ground-truth text transcript | GeminiEmbedding intent classification · Speech-MASSIVE, German and Arabic | accuracy: 0.84 to 0.83; 0.77 to 0.67 | Mixed |
| Use Whisper ASR + GeminiEmbedding instead of ground-truth text for query reranking | SVQ acoustic hypothesis reranking · SVQ locales | mAP: 0.86 mAP; 0.11 mAP | Mixed |
| Use raw spectrogram features instead of pre-trained HuBERT or Wav2Vec2 embeddings | SVQ speaker clustering · SVQ locales | V-measure: 0.97 V-measure for speaker ID in te-IN | Mixed |
| Use audio input instead of ground-truth transcript | Whisper Large v3 transcription · SVQ, English, clean vs background speech | WER: 1.6\% -> 3.8\% | Hurt |
Surprising findings
- Raw spectrogram features matched or outperformed HuBERT and Wav2Vec2 for speaker clustering on SVQ in many locales.
- Retrieval with sound input consistently lags text-oracle input, and ASR WER does not perfectly correlate with retrieval MRR.
Limitations
- SVQ is released as a single undivided evaluation set without train/validation/test splits, limiting training use.
- Initial release covers only eight task families and mostly speech-and-text modalities, with image and other modalities planned.
- Several baselines are cascades dependent on ASR quality, and some tasks rely on zero-shot evaluation without fine-tuning.
Experimental setup
Datasets: SVQ Speech-MASSIVE FSD50K BirdSet XTREME-UP MASSIVE
Benchmarks: MSEB retrieval MSEB reranking MSEB reasoning MSEB classification MSEB transcription MSEB segmentation MSEB clustering MSEB reconstruction FSD50K BirdSet Speech-MASSIVE SVQ
Metrics: MRR exact match mAP gmean-F1 accuracy WER CER NDCG V-measure FAD
Baselines: Whisper Large v3 cascade with GeminiEmbedding Gecko text embedding CLAP Perch EnCodec HuBERT Wav2Vec2 Gemma 3
Builds on: MTEB XTREME-UP Speech-MASSIVE FSD50K BirdSet
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.87)
- AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.87)
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.87)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.87)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.87)
- MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.