Research map / Vision-language and multimodal / CLIP and zero-shot
Pengi: An Audio Language Model for Audio Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on multiple tasks with shared encoder instead of captioning only (ablation) | Pengi · AudioCaps captioning | SPIDEr: 0.4551 -> 0.4667 | Helped |
| train on multiple tasks with shared encoder instead of captioning only (ablation) | Pengi · Clotho captioning | SPIDEr: 0.2648 -> 0.2709 | Helped |
| train the audio encoder instead of keeping it frozen (ablation) | Pengi · ESC50 zero-shot | accuracy: 0.8950 -> 0.9195 | Helped |
| train the audio encoder instead of keeping it frozen (ablation) | Pengi · US8K zero-shot | accuracy: 0.6319 -> 0.7185 | Helped |
| include text encoder before mapper m2 (ablation) | Pengi · Clotho captioning | SPIDEr: 0.2545 -> 0.2709 | Helped |
| include text encoder before mapper m2 (ablation) | Pengi · AudioCaps captioning | SPIDEr: 0.4857 -> 0.4667 | Hurt |
| remove text encoder and mapper m2 (text direct to LM) (ablation) | Pengi · emotion recognition | accuracy | Hurt |
| generative pretraining (Pengi) vs contrastive pretraining on same data (CLAP*) (ablation) | Pengi · ESC50 zero-shot | accuracy: 0.8916 -> 0.9195 | Helped |
| generative pretraining (Pengi) vs contrastive pretraining on same data (CLAP*) (ablation) | Pengi · US8K zero-shot | accuracy: 0.7661 -> 0.7185 | Hurt |
| use 'generate metadata' prompt instead of 'generate audio caption' (ablation) | Pengi · ESC50 zero-shot | accuracy: 0.8870 -> 0.9195 | Helped |
| use 'generate metadata' prompt instead of 'generate audio caption' (ablation) | Pengi · US8K zero-shot | accuracy: 0.7185 -> 0.6585 | Hurt |
| use three linear layers instead of one on frozen Pengi encoder (ablation) | Pengi · ESC50 | accuracy: 0.8915 -> 0.9485 | Helped |
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.89)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.89)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.88)
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.