Research map / Vision-language and multimodal / Human and embodied modeling
VLGrammar: Grounded Grammar Induction of Vision and Language
Yining Hong, Qing Li, Song-Chun Zhu, Siyuan Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SCAN pretraining of the part clustering module (ablation) | VLGrammar · PartIt image grammar (all categories) | corpus-level F1 / instance-level F1: 44.7 -> 51.4 corpus, 55.5 -> 63.4 instance | Improved |
| add SCAN pretraining of the part clustering module (ablation) | VLGrammar · PartIt part clustering (all categories) | accuracy: 64.4 -> 69.1 | Improved |
| add SCAN pretraining of the part clustering module (ablation) | VLGrammar · PartIt image grammar (table) | corpus-level F1 / instance-level F1: 57.9 -> 46.3 corpus, 75.4 -> 60.5 instance | Hurt |
| add SCAN pretraining of the part clustering module (ablation) | VLGrammar · PartIt language grammar (all categories) | corpus-level F1 / instance-level F1: 49.0 -> 51.3 corpus, 49.8 -> 51.9 instance | No clear effect |
| train image grammar jointly with language grammar (VLGrammar vs V-PCFG) | V-PCFG · PartIt part clustering (all categories) | accuracy: 41.3 -> 69.1 (SCAN to VLGrammar) | Improved |
| joint contrastive training of vision and language grammars (vs V-PCFG) | V-PCFG · PartIt image grammar (all categories) | corpus-level F1: 47.5 -> 51.4 | Improved |
| joint training vs language grounding only (vs L-PCFG) | L-PCFG · PartIt language grammar (all categories) | corpus-level F1: 48.4 -> 51.3 | Improved |
| pretrain a category-agnostic language PCFG then fine-tune per category | L-PCFG · PartIt language grammar (all categories) | corpus-level F1: 47.8 -> 48.4 | Improved |
| add grammar induction to a ResNet-18/BERT retrieval model | retrieval baseline · PartIt image-text retrieval (chair) | text-to-image accuracy: 24.1 -> 33.2 IR, 28.5 -> 39.0 TR | Improved |
| add grammar induction to a ResNet-18/BERT retrieval model | retrieval baseline · PartIt image-text retrieval (bed) | text-to-image accuracy: 20.1 -> 39.6 IR, 20.1 -> 38.2 TR | Improved |
| use language-grounded training (V-PCFG-LG) vs VLGrammar on unseen categories | V-PCFG-LG · PartIt unseen category bed (trained on chair+table) | instance-level F1: 50.4 -> 44.2 | Hurt |
| train image grammar on chair+table and test on unseen category | VLGrammar · PartIt unseen category bed | corpus-level F1: 20.7 -> 29.4 (V-PCFG to VLGrammar) | Improved |
Closest papers by meaning
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.89)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.88)
- FRABench and UFEval: Unified Fine-grained Evaluation with Task and Aspect Generalization (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
