Research map / Vision-language and multimodal / CLIP and zero-shot
Querying as Prompt: Parameter-Efficient Learning for Multimodal Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QaP, a parameter-efficient multimodal language model strategy that feeds learnable modality-specific querying prompts into a frozen DeBERTa-V2-XLarge with lightweight text-conditioned resamplers and FFN adapters, reaching state-of-the-art on Music-AVQA, How2QA, TVQA and CMU-MOSEI with about 4.6% trainable parameters.
Problem
Efficiently bridging multiple modalities (audio, video, text) into pre-trained language models for multimodal tasks, without the long input sequences of projection-based methods or the heavy external resamplers (e.g., Q-Former) of querying-based methods.
Approach
Treat learnable modality-specific querying prompts as both queries for extracting modality information and soft prompts inside the LM; insert lightweight gated text-conditioned resamplers (a linear projection plus parameter-free dot-product cross-attention) into LM layers so that text-relevant multimodal information is injected layer by layer.
Key results
- Achieves 78.41% accuracy on Music-AVQA with 40M trainable parameters, versus 77.17% for Lavish (21.09M) and 77.73% for full fine-tuning (890M).
- Reaches 94.5% on How2QA, above Frozenbilm's 86.7%, and 80.19% on TVQA validation versus Frozenbilm's 82.4%.
- On CMU-MOSEI, reports Corr 0.825 versus 0.773 for UniMSE, the strongest baseline on that metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QaP instead of Lavish | Multimodal language model · Music-AVQA | accuracy: 77.17% -> 78.41% | Helped |
| Use QaP instead of Frozenbilm | VideoQA model · How2QA | accuracy: 86.7% -> 94.5% | Helped |
| Use QaP instead of Frozenbilm | VideoQA model · TVQA val | accuracy: 82.4% -> 80.19% | Hurt |
| Use QaP instead of UniMSE | Multimodal sentiment model · CMU-MOSEI | MAE: 0.523 -> 0.529 | Hurt |
| Use QaP instead of UniMSE | Multimodal sentiment model · CMU-MOSEI | Corr: 0.773 -> 0.825 | Helped |
| Use QaP instead of UniMSE | Multimodal sentiment model · CMU-MOSEI | ACC-2: 85.86/87.50 -> 86.95/88.03 | Helped |
| Use QaP instead of UniMSE | Multimodal sentiment model · CMU-MOSEI | F1: 85.79/87.46 -> 90.87/90.90 | Helped |
| Use a shared Key and Value projection rather than separate projected Query, Key, or Value variants (ablation) | Text-Conditioned Resampler on Music-AVQA · Music-AVQA | accuracy: 76.14% -> 78.41%; 77.53% -> 78.41%; 78.01% -> 78.41% | Helped |
| Insert TCR in layers 1-12 rather than layers 12-24, 1-24, or even-indexed layers (ablation) | QaP on Music-AVQA · Music-AVQA | accuracy: 76.64% -> 78.41%; 77.56% -> 78.41%; 77.98% -> 78.41%; 76.82% -> 78.41% | Helped |
| Initialize gates to 1 in the first 6 layers and 0 in the subsequent 6 rather than removing the gate or using other initializations (ablation) | QaP on Music-AVQA · Music-AVQA | accuracy: 78.07% -> 78.41%; 78.32% -> 78.41%; 76.93% -> 78.41%; 76.66% -> 78.41% | Mixed |
| Use 1, 2, or 5 modality-specific querying prompts; avoid sharing prompts between modalities or directly concatenating multimodal features (ablation) | QaP on Music-AVQA · Music-AVQA | accuracy | Mixed |
Surprising findings
- Initializing all TCR gates to 0 or applying tanh to the gates significantly reduces accuracy, suggesting joint multimodal information is needed in early layers.
- Inserting the resampler only in layers 12-24 still beats the reimplemented eP-ALM by about 1.5%, but inserting in layers 1-12 performs best.
Limitations
- Only validated on bidirectional (encoder-style) language models; decoder-only autoregressive LLMs not explored.
- Not yet tested with larger LLMs or additional modalities such as point clouds and depth maps.
- Could not access TVQA test labels, so results are reported on the validation set.
Experimental setup
Datasets: Music-AVQA How2QA TVQA CMU-MOSEI
Benchmarks: Music-AVQA How2QA TVQA CMU-MOSEI
Metrics: accuracy MAE Pearson correlation Acc-2 F1
Baselines: Lavish AVQA PSTP-Net Limber PromptFuse MAGMA eP-ALM Frozenbilm UniMSE Self-MM MMIM MulT
Builds on: Frozen BiLM / FrozenBiLM Parallel Adapter Q-Former Flamingo eP-ALM LoRA
Closest papers by meaning
- Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (ICLR 2026 · similarity 0.88)
- Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · similarity 0.86)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.86)
- ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.86)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.