Research map / Large language models / Language, safety and interpretability
Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use raw wav2vec 2.0 speech representations instead of wav2vec 2.0 transcriptions | Grounded language model · GoLD object retrieval | Triplet MRR: 0.83 ($\pm$0.002) -> 0.85 ($\pm$0.002) | Helped |
| Use raw wav2vec 2.0 speech representations instead of wav2vec 2.0 transcriptions | Grounded language model · GoLD object retrieval | Subset MRR: 0.83 ($\pm$0.002) -> 0.86 ($\pm$0.002) | Helped |
| Use raw wav2vec 2.0 speech representations instead of wav2vec 2.0 transcriptions | Grounded language model · GoLD object retrieval | F1: 0.79 ($\pm$0.003) -> 0.83 ($\pm$0.003) | Helped |
| Use wav2vec 2.0 representations instead of vq-wav2vec, DeCoAR, or MFCC features | Raw-speech grounded language model · GoLD object retrieval | Subset MRR: vq-wav2vec 0.78; DeCoAR 0.72; MFCC 0.49; wav2vec 2.0 Speech 0.86 | Helped |
| Use direct speech grounding rather than transcription-first grounding | wav2vec 2.0 grounded language model · Individual users with accented speech | Subset MRR difference between accented and non-accented users: approximately 6% vs 2% | Helped |
| Use raw speech rather than transcriptions | wav2vec 2.0 model trained on user groups · GoLD user groups, accent | Subset MRR: 0.92 -> 0.93 | Helped |
| Use raw speech rather than transcriptions | wav2vec 2.0 model trained on user groups · GoLD user groups, accent | Triplet MRR: 0.89 -> 0.90 | Helped |
| Use raw speech rather than transcriptions | wav2vec 2.0 model trained on user groups · GoLD user groups, non-accent | Triplet MRR: 0.85 -> 0.85 | No clear effect |
| Use MFCC features instead of pretrained speech representations | Grounded language model · GoLD object retrieval | Subset MRR: 0.86 -> 0.49 | Hurt |
Closest papers by meaning
- A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning (NeurIPS 2021 · similarity 0.90)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.86)
- Learning Language Structures Through Grounding (AAAI 2025 · similarity 0.86)
- GROUNDHOG: Grounding Large Language Models to Holistic Segmentation (CVPR 2024 · similarity 0.85)
- Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data (NeurIPS 2025 · similarity 0.85)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.