Research map / Vision-language and multimodal / CLIP and zero-shot
CoLLAT: On Adding Fine-grained Audio Understanding to Language Models using Token-Level Locked-Language Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add token-level cross-modal alignment loss (Lcross-token) (ablation) | CoLLAT · FSD50K zero-shot, clean | MAP: 0.13 -> 0.19 | Helped |
| add token-level cross-modal alignment loss (Lcross-token) (ablation) | CoLLAT · ESC-50 zero-shot, clean | accuracy: 0.81 -> 0.84 | Helped |
| add token-level cross-modal alignment loss (Lcross-token) (ablation) | CoLLAT · ESC-50 zero-shot, noisy | accuracy: 0.50 -> 0.59 | Helped |
| add unimodal token-level loss (Luni-token) (ablation) | CoLLAT · ESC-50 zero-shot, noisy | accuracy: 0.53 -> 0.59 | Helped |
| add unimodal token-level loss (Luni-token) (ablation) | CoLLAT · FSD50K zero-shot, noisy | MAP: 0.13 -> 0.17 | Helped |
| add unimodal token-level loss (Luni-token) (ablation) | CoLLAT · ESC-50 zero-shot, clean | accuracy: 0.85 -> 0.84 | No clear effect |
| add cross-modal global contrastive loss (Lcross-global) (ablation) | CoLLAT · ESC-50 zero-shot, clean | accuracy: 0.82 -> 0.84 | Helped |
| add cross-modal global contrastive loss (Lcross-global) (ablation) | CoLLAT · FSD50K zero-shot, noisy | MAP: 0.16 -> 0.17 | Helped |
| use pretrained AST audio encoder instead of ResNet (ablation) | CoLLAT · ESC-50 zero-shot | accuracy: 0.80 -> 0.84 | Helped |
| use pretrained AST audio encoder instead of ResNet (ablation) | CoLLAT · FSD50K zero-shot | MAP: 0.14 -> 0.19 | Helped |
| pretrain AST audio encoder on AudioSet instead of random initialization (ablation) | CoLLAT · ESC-50 zero-shot | accuracy: 0.82 -> 0.84 | Helped |
| CoLLAT with locked text encoder and token alignment vs CLAP | CoLLAT · AudioSet zero-shot | MAP: 0.06 -> 0.09 | Helped |
Closest papers by meaning
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.89)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.89)
- MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks (ICML 2026 · similarity 0.89)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.88)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.87)
- UniAudio 1.5: Large Language Model-Driven Audio Codec is A Few-Shot Audio Task Learner (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.