Research map / Vision-language and multimodal / Cross-modal learning
LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues
Amir Ivry, Shinji Watanabe
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio to transcript input (ablation) | Qwen2-Audio · specificity (GT transcripts) | Specificity: 0.667 -> 0.800 | Improved |
| Add audio to transcript input (ablation) | Qwen2-Audio · sensitivity (GT transcripts) | Sensitivity: 0.229 -> 0.231 | No clear effect |
| Use audio-only instead of transcript input (ablation) | Audio Flamingo 3 · sensitivity | Sensitivity: 0.28 -> 0.154 | Hurt |
| Replace GT transcripts with Whisper-Large (ablation) | LLaMA-3.1-8B judge · sensitivity | Sensitivity: 0.457 -> 0.282 | Hurt |
| Replace GT transcripts with Whisper-Large (ablation) | LLaMA-3.1-8B judge · specificity | Specificity: 0.941 -> 0.924 | No clear effect |
| Replace GT with Whisper transcripts (ablation) | Qwen2-Audio transcript-only · specificity | Specificity: 0.667 -> 0.685 | Improved |
| Add audio to transcripts (ablation) | Judges across categories · category-level specificity | Specificity: hate 0.54 -> 0.71; deception 0.61 -> 0.76 | Improved |
| Choose Sens-optimal vs Spec-optimal prompt (ablation) | 24 configurations · sensitivity/specificity | Sens and Spec: Sens range up to 0.4, Spec up to 0.27 | Mixed |
| Add audio to transcripts (ablation) | Qwen2-Audio · position bias (Sens) | AAPB Sens: 0.029 -> 0.294 | Hurt |
| Add audio to transcripts (ablation) | MiniCPM-o-4.5 · sensitivity | Sensitivity: -0.126 -> -0.109 | No clear effect |
| Add audio to transcripts (ablation) | Qwen2.5-Omni-7B · specificity | Specificity: +0.3 -> 0 | Hurt |
Closest papers by meaning
- LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models (CVPR 2026 · similarity 0.89)
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics (ICLR 2025 · similarity 0.88)
- Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models (ICLR 2026 · similarity 0.87)
- Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models (NeurIPS 2025 · similarity 0.87)
- ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio–Language Models (NeurIPS 2025 · similarity 0.87)
- EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
