Research map / Vision-language and multimodal / Cross-modal learning

Hierarchical Aligned Multimodal Learning for NER on Tweet Posts

AAAI 2024 · Vision-language and multimodal › Cross-modal learning · approach: Probabilistic embeddings for cross-modal retrieval · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HamLearning, an end-to-end hierarchical multimodal NER model that measures text-image relevance and iteratively aligns global/local visual features with words, evaluated on Twitter-2015 and Twitter-2017.

Problem

Multimodal named entity recognition on tweets suffers from noisy or irrelevant images and neglects implicit visual scene knowledge and multi-level vision-text alignment.

Approach

Combines intra-modality encoding (BERT+Transformer for text; ViT semantic and R-GCN spatial graph over detected objects for vision), a learned text-image relevance score that gates global visual features without auxiliary relevance-classification tasks, and iterative cross-modal Transformer co-attention before CRF decoding.

Key results

• HamLearning reports overall F1 of 76.49 on Twitter-2015 and 87.13 on Twitter-2017, exceeding the listed SOTA baselines.
• Cross-dataset generalization: 67.98 F1 (Twitter-17 to Twitter-15) and 64.70 F1 (Twitter-15 to Twitter-17), well above UMT and UMGF.
• Removing the relevance measuring module drops F1 to 74.61 on Twitter-2015 and 85.60 on Twitter-2017.

Measured effects reported in the paper

Surprising findings

• Removing the R-GCN spatial encoder hurts more than removing the ViT semantic encoder, suggesting spatial object relations matter more than semantic encoding.

Limitations

• Relies on Faster R-CNN object detection; Mask R-CNN with COCO gives slightly lower performance.
• Cross-modal iteration depth L beyond 3 degrades performance, indicating sensitivity to interaction depth.

Experimental setup

Datasets: TWITTER-2015 TWITTER-2017 Visual Genome MS COCO

Benchmarks: TWITTER-2015 TWITTER-2017

Metrics: precision recall F1

Baselines: BiLSTM-CRF CNN-BiLSTM-CRF HBiLSTM-CRF BERT-CRF AdaCAN-BERT-CRF GVATT-BERT-CRF UMT UMGF MAF RDS GEI ITA

Builds on: BERT R-GCN Faster R-CNN CRF decoding

Closest papers by meaning

• Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (NeurIPS 2024 · similarity 0.86)
• Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition (AAAI 2025 · similarity 0.86)
• A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.86)
• Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.86)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.85)
• MCL-NER: Cross-Lingual Named Entity Recognition via Multi-View Contrastive Learning (AAAI 2024 · similarity 0.84)