Research map / Vision-language and multimodal / Human and embodied modeling
Learning Language Structures Through Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A PhD thesis that proposes learning language structures (phrase, dependency, CCG/programs, word alignment) from distant "grounds" such as vision, program execution, and bitext in other languages.
Problem
Learning syntactic and semantic language structures without explicit annotation, using indirect supervision from other data sources (grounds) such as images, program execution results, and parallel sentences in other languages.
Approach
Formalizes "grounding" as linking a primary text source to a paired ground source (vision, execution results, other-language text) and uses it as indirect supervision to induce phrase structures, CCG/programs, word alignments, and dependency parses.
Key results
- VG-NSL+AI achieves average F1 of 53.3 on MSCOCO, above PRPN (52.5) and ON-LSTM (45.5) among models not requiring extra labels.
- VG-NSL+AI has self-agreement F1 of 90.2 versus 60.3 for PRPN across random initializations.
- VG-NSL is reported to reach comparable performance using only 20% of training captions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use visual grounding to induce constituency parses | VG-NSL · MSCOCO test | Avg. F1: 52.5 -> 53.3 | Helped |
| add the abstract-initial inductive bias (ablation) | VG-NSL · MSCOCO test | Avg. F1: 50.4 -> 53.3 | Helped |
| add the abstract-initial inductive bias (ablation) | VG-NSL · MSCOCO test | PP recall: 42.0 -> 66.5 | Helped |
| add the abstract-initial inductive bias (ablation) | VG-NSL · MSCOCO test | NP recall: 79.6 -> 74.6 | Hurt |
| add the abstract-initial inductive bias (ablation) | VG-NSL · MSCOCO test | VP recall: 26.2 -> 32.5 | Helped |
| add the abstract-initial inductive bias (ablation) | VG-NSL · MSCOCO test | Self F1: 87.1 -> 90.2 | Helped |
| use pre-trained FastText word embeddings (ablation) | VG-NSL + abstract-initial bias · MSCOCO test | Avg. F1: 53.3 -> 54.4 | Helped |
| use pre-trained FastText word embeddings (ablation) | VG-NSL + abstract-initial bias · MSCOCO test | VP recall: 32.5 -> 24.4 | Hurt |
| use pre-trained FastText word embeddings (ablation) | VG-NSL + abstract-initial bias · MSCOCO test | Self F1: 90.2 -> 89.8 | Hurt |
Surprising findings
- ON-LSTM, which is state-of-the-art on WSJ, performs poorly on MSCOCO captions (avg F1 45.5).
Limitations
- Evaluation limited to MSCOCO captions, not abstract domains like WSJ.
- Abstract-initial inductive bias is language-specific and does not apply to head-final languages.
Experimental setup
Datasets: MSCOCO Multi30K Penn Treebank
Benchmarks: MSCOCO test split (Benepar-parsed gold trees)
Metrics: bracket F1 self-agreement F1 recall of NP/VP/PP/ADJP
Baselines: PRPN ON-LSTM Gumbel tree PMI random trees left-branching right-branching
Builds on: VG-NSL PRPN ON-LSTM Gumbel tree composition
Closest papers by meaning
- Systematic Generalization: What Is Required and Can It Be Learned? (ICLR 2019 · similarity 0.86)
- Mapping Language Models to Grounded Conceptual Spaces (ICLR 2022 · similarity 0.86)
- Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech (AAAI 2022 · similarity 0.86)
- Grammar-Based Grounded Lexicon Learning (NeurIPS 2021 · similarity 0.86)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.85)
- Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.