Research map / Video / Video segmentation and tracking
MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTGA, a multi-view event-based lip-reading model that fuses event frames with a time-segmented voxel graph list via temporal granularity aligned fusion and a positional-encoding Bi-GRU/self-attention back-end, reaching 75.08% on DVS-Lip.
Problem
Event-based lip-reading loses fine-grained intra-frame temporal information when events are aggregated into event frames.
Approach
Represent events from two views, global event frames and a time-segmented voxel graph list of the most significant local voxels connected by GMMConv, align them temporally per segment, fuse them iteratively, and aggregate with positional encoding plus Bi-GRU and self-attention.
Key results
- On DVS-Lip, MTGA reaches 75.08% overall accuracy versus 72.10% for MSTP (the strongest baseline in the table).
- On DVS128-Gait-Day at 10 epochs, MTGA reaches 95.42% versus 90.28% for MSTP.
- Ablation: the voxel graph list branch alone reaches 70.03% and the event frame branch alone 71.71%; the fused model reaches 75.08%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MTGA instead of MSTP | event-based lip-reading · DVS-Lip test | Acc (%): 72.10 -> 75.08 | Helped |
| use MTGA instead of MSTP | event action recognition · DVS128-Gait-Day | Acc (%): 90.28 -> 95.42 | Helped |
| add the voxel graph list branch to the event frame branch (ablation) | MTGA · DVS-Lip test | Acc (%): 71.71 -> 75.08 | Helped |
| add the event frame branch to the voxel graph list branch (ablation) | MTGA · DVS-Lip test | Acc (%): 70.03 -> 75.08 | Helped |
| replace fixed-weight fusion with temporal granularity aligned fusion (ablation) | MTGA · DVS-Lip test | Acc (%): 71.26 -> 75.08 | Helped |
| replace attention fusion with temporal granularity aligned fusion (ablation) | MTGA · DVS-Lip test | Acc (%): 72.51 -> 75.08 | Helped |
| add position embedding to the Bi-GRU backend (ablation) | MTGA · DVS-Lip test | Acc (%): 73.56 -> 74.15 | Helped |
| add self-attention to the Bi-GRU backend (ablation) | MTGA · DVS-Lip test | Acc (%): 73.56 -> 74.87 | Helped |
| add position embedding and self-attention to the Bi-GRU backend (ablation) | MTGA · DVS-Lip test | Acc (%): 73.56 -> 75.08 | Helped |
Limitations
- Evaluated on only one lip-reading dataset (DVS-Lip) plus one gait dataset.
- Voxel graph branch alone is weaker than the event frame branch.
- Hardware and training details are not described in the text.
Experimental setup
Datasets: DVS-Lip DVS128-Gait-Day
Benchmarks: DVS-Lip DVS128-Gait-Day
Metrics: accuracy Acc1 Acc2
Baselines: Event Clouds EST ACTION-Net Martinez et al. MS-TCN AGCN GET MSTP
Builds on: MSTP GMMConv ResBlock Bi-GRU
Closest papers by meaning
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.85)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.85)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.84)
- Audio-Visual Semantic Graph Network for Audio-Visual Event Localization (CVPR 2025 · similarity 0.84)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.84)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.