Research map / Graph learning / Knowledge and temporal graphs
PHGC: Procedural Heterogeneous Graph Completion for Natural Language Task Verification in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PHGC, a heterogeneous graph method with graph construction, cross-modal path finding, and discriminative entity aggregation for natural-language egocentric task verification, and releases the CSV-NL real-video benchmark.
Problem
Natural language-based egocentric task verification (NLETV) must check whether egocentric videos follow textual procedural rules, facing cross-modal heterogeneity between text rules and video, and hierarchical misalignment between rule logic and operation flows.
Approach
Represent rules and video as a heterogeneous graph of hierarchical vertices (object states, operations, rule, visual actions, visual progress), add learned cross-modal matching edges via a DP-based heuristic edge-cutting strategy, and aggregate the final discriminative entity vertex with graph attention to make the binary verification decision.
Key results
- On EgoTV, PHGC reaches 87.4 average F1 versus 80.0 for the prior NSG method.
- On CSV-NL, PHGC reaches 68.49 average F1 versus 61.04 for NSG.
- Removing the HGC module drops EgoTV average F1 from 91.7-style full-model results to 61.1-level, indicating the graph construction is essential (ablation shows large drops across all settings).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Heterogeneous Graph Construction module (ablation) | PHGC · EgoTV novel steps | F1-score: 50.7 -> 58.2 | Helped |
| Add the Cross-Modal Path Finding module (ablation) | PHGC with HGC · EgoTV novel steps | F1-score: 58.2 -> 62.6 | Helped |
| Add the Discriminative Entity Representation module (ablation) | PHGC with HGC and CMPF · EgoTV novel steps | F1-score: 62.6 -> 82.4 | Helped |
| Add the Heterogeneous Graph Construction module (ablation) | PHGC · CSV-NL average | F1-score: 54.8 -> 60.0 | Helped |
| Add the Cross-Modal Path Finding module (ablation) | PHGC with HGC and DER · CSV-NL average | F1-score: 62.8 -> 68.1 | Helped |
| Add the Discriminative Entity Representation module (ablation) | PHGC with HGC and CMPF · CSV-NL average | F1-score: 59.8 -> 68.1 | Helped |
| Remove edges from the discriminative entity to state vertices (ablation) | PHGC · EgoTV novel steps | F1-score: 82.4 -> 69.6 | Hurt |
| Remove edges from the discriminative entity to rule vertices (ablation) | PHGC · CSV-NL average | F1-score: 68.0 -> 62.2 | Hurt |
Surprising findings
- Multimodal LLMs such as GoldFish and PandaGPT remain competitive on the realistic CSV-NL benchmark, with PandaGPT and GoldFish at 57-59 average F1, while the gain of PHGC is smaller there than on EgoTV.
Limitations
- Performance gain is smaller on the realistic CSV-NL dataset than on synthetic EgoTV.
- Relies on frozen CLIP features and T5-derived textual graph vertices extracted offline.
- Evaluation limited to two benchmarks and the F1 metric.
Experimental setup
Datasets: EgoTV CSV-NL
Benchmarks: EgoTV CSV-NL
Metrics: F1 score
Baselines: NSG PandaGPT GoldFish Otter VIOLIN MIL-NCE CLIP CoCa VideoCLIP CLIP4Clip
Builds on: NSG (Rishi et al., ICCV 2023) CLIP T5 graph attention network
Closest papers by meaning
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos (NeurIPS 2024 · similarity 0.87)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.85)
- Error Detection in Egocentric Procedural Task Videos (CVPR 2024 · similarity 0.84)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.84)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.83)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.