Research map / Vision-language and multimodal / Visual question answering
Mask4Align: Aligned Entity Prompting with Color Masks for Multi-Entity Localization Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mask4Align overlays color-coded masks on a grayscale scene and uses candidate color-entity answers with pretrained VQA models (ALBEF, CLIP) to localize one entity among many similar ones, without training.
Problem
Localizing a specific entity among multiple same-category entities in visual question answering is hard because text descriptions cannot distinguish similar entities.
Approach
Convert the scene to grayscale, paint each candidate entity mask with a distinct color, and pose the question with color-aligned candidate answers ("the {Color} {Entity}") to a pretrained VQA module; for more than six entities, recursively group masks with k-means and repeat the VQA process.
Key results
- On ME-VQA, ALBEF + Mask4Align reaches S-measure 0.821 and MAE 0.071 versus 0.514 and 0.342 for ALBEF + Random.
- On GazeFollow, ALBEF + Mask4Align improves AUC from 0.688 to 0.745 and angular error from 39.2 to 32.4 over ALBEF alone.
- Mask4Align outperforms the Box4Align and Covering4Align variants by about 0.020 AUC on GazeFollow.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add color masks and aligned candidate answers | ALBEF · GazeFollow test set | AUC: 0.688 -> 0.745 | Helped |
| Add color masks and aligned candidate answers | ALBEF · ME-VQA | Sα: 0.514 -> 0.821 | Helped |
| Add RG-VQA (ablation) | ALBEF + Mask4Align · GazeFollow test set | AUC: 0.742 -> 0.745 | Helped |
| Add RG-VQA (ablation) | ALBEF + Mask4Align · ME-VQA | Sα: 0.781 -> 0.821 | Helped |
| Vary α from 0.3 to 0.7 (ablation) | CLIP + Mask4Align · GazeFollow test set | AUC: 0.674; 0.674; 0.675; 0.676; 0.673 | No clear effect |
| Vary α from 0.3 to 0.7 (ablation) | ALBEF + Mask4Align · GazeFollow test set | AUC: 0.742; 0.742; 0.742; 0.741; 0.740 | No clear effect |
| Replace color masks with a black bounding box (ablation) | CLIP · GazeFollow test set | AUC: 0.675 -> 0.655 | Hurt |
| Replace color masks by covering other entities with rectangles (ablation) | CLIP · GazeFollow test set | AUC: 0.675 -> 0.653 | Hurt |
Surprising findings
- Pretrained VQA models tend to prefer conventional color-object descriptions, e.g., choosing 'red fruit' over the correct 'orange fruit'.
Limitations
- Performance depends directly on the underlying VQA model.
- Best-performing VLMs do not accept textual candidate answers, limiting applicability.
- Color set is predefined and not adapted to model preferences; only six colors, requiring recursive grouping for more entities.
Experimental setup
Datasets: GazeFollow VQA
Benchmarks: GazeFollow ME-VQA
Metrics: AUC Dist Ang S-measure F-measure MAE
Baselines: CLIP ALBEF CLIP + Random ALBEF + Random Box4Align Covering4Align
Builds on: ALBEF CLIP Recognize Anything Model (RAM) SAM CPT
Closest papers by meaning
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.87)
- Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding (CVPR 2025 · similarity 0.87)
- Towards Robust Visual Question Answering via Prompt-Driven Geometric Harmonization (AAAI 2025 · similarity 0.85)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.84)
- Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs (CVPR 2024 · similarity 0.84)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.