Research map / Large language models / Language, safety and interpretability
BFCL Audio: An Audio Function Calling Evaluation for Large Language Models
Huanzhi Mao, Aditya Ghai, Imra Dawoodani, Tony A Ginart, Shishir G Patil, John Emmons, Joseph E. Gonzalez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add background noise to speech input (ablation) | GPT-4o-audio · BFCL True Audio overall | overall accuracy: 60.4 -> 50.9 | Hurt |
| add background noise to speech input (ablation) | Qwen3-Omni-Flash · BFCL True Audio overall | overall accuracy: 55.8 -> 48.2 | Hurt |
| add background noise to speech input (ablation) | Gemini-2.5-Pro · BFCL True Audio overall | overall accuracy: 51.1 -> 45.4 | Hurt |
| add competing speech (double-talk) (ablation) | clean utterances · intent preservation | intent preservation: 42.4% | Hurt |
| add device artifacts (mic rubbing, mumbling, fades, gain changes) (ablation) | clean utterances · intent preservation | intent preservation: 98.8% (mic rubbing), 100.0% (mumbling) | No clear effect |
| inject discourse-altering disfluencies (self-corrections, ellipsis, restarts) (ablation) | ASR->LLM pipeline · BFCL Text Audio | downstream failures | Hurt |
| inject surface-level disfluencies (fillers, repetitions, contractions) (ablation) | ASR->LLM pipeline · BFCL Text Audio | downstream failures | No clear effect |
| verbalize file paths in speech (ablation) | end-to-end audio models · BFCL True Audio (clean) | errors | Hurt |
| use end-to-end audio input instead of ASR transcripts (clean audio) | GPT-4o-audio · BFCL Audio overall | overall accuracy: 58.6 -> 60.4 | Improved |
| use end-to-end audio input instead of ASR transcripts (noisy audio) | GPT-4o-audio · BFCL Audio overall | overall accuracy: 58.6 -> 50.9 | Hurt |
Closest papers by meaning
- The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (ICML 2025 · similarity 0.88)
- $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains (ICML 2026 · similarity 0.88)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.86)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.86)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.86)
- Robust Function-Calling for On-Device Language Model via Function Masking (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
