Research map / Vision-language and multimodal / Visual question answering
SkillCLIP: Skill Aware Modality Fusion Visual Question Answering (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkillCLIP, a skill-aware late-fusion VQA model that predicts visual skills for a question and fuses CLIP encodings with expert-module outputs (object detection, OCR, skill embeddings), evaluated on VizWiz VQA.
Problem
VizWiz VQA is hard due to blurry, out-of-frame, conversational questions from visually impaired users, and pre-trained VQA architectures transfer poorly; without huge multimodal LLMs, it is unclear how to decompose questions into skills and integrate expert outputs.
Approach
Predict the visual skills needed for a question (skill embedding) and use it to guide late fusion of independently encoded CLIP image/question features with expert module outputs (DETA object tags, EasyOCR scene text), plus a multitask variant that jointly learns skill prediction and answering.
Key results
- SkillCLIP reaches 62.17% dev accuracy vs. 61.82% for ViLT and 51.24% for GIT-Large; BERT-sim answer score 68.6 vs. 67.1 for ViLT.
- SkillCLIP improves text-skill accuracy to 39.16% (vs. 36.05% ViLT) and count-skill accuracy to 50.45% (vs. 41.44% ViLT).
- Ablations show gains are modest; SkillCLIP-multitask (62.16%) and ablated variants perform similarly to the full model on overall dev accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use skill-aware fusion instead of the strongest listed baseline, ViLT | SkillCLIP on VizWiz · VizWiz dev | Answer BERTSim (λ = 0.7): 67.122% -> 68.627% | Helped |
| Use skill-aware fusion instead of the strongest listed baseline, ViLT | SkillCLIP on VizWiz · VizWiz dev | Dev accuracy: 61.82% -> 62.17% | Helped |
| Add skill embeddings to the fusion model (ablation) | SkillCLIP · VizWiz dev | Answer BERTSim (λ = 0.7): 68.743% -> 68.627% | Hurt |
| Add skill embeddings to the fusion model (ablation) | SkillCLIP · VizWiz dev | Count accuracy: 45.95% -> 50.45% | Helped |
| Add object tags (ablation) | SkillCLIP · VizWiz dev | Answer BERTSim (λ = 0.7): 69.553% -> 68.627% | Hurt |
| Add object tags (ablation) | SkillCLIP · VizWiz dev | Count accuracy: 45.95% -> 50.45% | Helped |
| Add scene text (ablation) | SkillCLIP · VizWiz dev | Answer BERTSim (λ = 0.7): 69.160% -> 68.627% | Hurt |
| Add scene text (ablation) | SkillCLIP · VizWiz dev | Text accuracy: 40.44% -> 39.16% | Hurt |
| Jointly learn skill prediction and answer classification (ablation) | SkillCLIP Multitask · VizWiz dev | Answer BERTSim (λ = 0.7): 68.627% -> 68.025% | Hurt |
Surprising findings
- Skill-aware fusion helps only for a subset of question types; overall dev accuracy gains over skill-unaware ablations are marginal (e.g., w/o skill embedding 62.48% vs. full 62.17%).
Limitations
- Gains over skill-unaware baselines are small and only for some skills.
- Relies on separate expert modules (OCR, object detector) whose errors propagate; late fusion with separately encoded modalities.
- Uses only four skill categories from VizWiz Vision Skills.
Experimental setup
Datasets: VizWiz VQA VizWiz Vision Skills COCO
Benchmarks: VizWiz VQA dev set
Metrics: accuracy per-skill accuracy (text, object, color, count) BERTScore similarity
Baselines: BERTSim CLIP-Large fusion ViLT GIT-Large
Builds on: CLIP ViLT GIT
Closest papers by meaning
- Separating Skills and Concepts for Novel Visual Question Answering (CVPR 2021 · similarity 0.91)
- Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (CVPR 2017 · similarity 0.90)
- Knowledge Acquisition for Visual Question Answering via Iterative Querying (CVPR 2017 · similarity 0.90)
- Roses Are Red, Violets Are Blue... but Should VQA Expect Them To? (CVPR 2021 · similarity 0.89)
- HybridPrompt: Bridging Language Models and Human Priors in Prompt Tuning for Visual Question Answering (AAAI 2023 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.