Research map / Graph learning / Knowledge and temporal graphs
SignGraph: A Sign Sequence is Worth Graphs of Nodes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SignGraph, a graph-based continuous sign language recognition model that represents each frame as a graph of patch nodes and builds intra-frame (LSG) and inter-frame (TSG) dynamic graphs in a multiscale design.
Problem
CNN-based backbones used in continuous sign language recognition treat frames as grids and fail to capture cross-region correlations within a frame and interactions across regions in adjacent frames.
Approach
Treat sign frames as sets of patch nodes and dynamically build a dense KNN graph within each frame (Local Sign Graph) and a sparse top-K node-pair graph between adjacent frames (Temporal Sign Graph), stacked in a multiscale pyramid, instead of using CNN convolutions.
Key results
- On PHOENIX14T, MultiSignGraph reaches 17.8% dev WER and 19.1% test WER, outperforming TwoStream's 19.3% test WER without extra cues.
- On CSL-Daily, MultiSignGraph achieves 27.3% dev and 26.4% test WER, outperforming most prior methods and comparable to TwoStream.
- In ablations on PHOENIX14T dev, the single-stage LSG+TSG reduces WER from 22.3% to 18.4%, and the graph-based backbone beats CvT, Swin, and PyVIG backbones (17.9% vs 39.2%, 45.4%, 35.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Local Sign Graph (LSG) module on ResNet18 baseline (ablation) | SignGraph single stage · PHOENIX14T dev | WER: decreases by 3.1% (LSG1) | Helped |
| add Temporal Sign Graph (TSG) module on ResNet18 baseline (ablation) | SignGraph single stage · PHOENIX14T dev | WER: decreases by 2.7% (TSG1) | Helped |
| combine LSG and TSG modules (ablation) | SignGraph · PHOENIX14T dev | WER: decreases by 3.9% (LSG1+TSG1) | Helped |
| two stages of LSG and TSG (multiscale) (ablation) | SignGraph · PHOENIX14T dev/test | WER: decreases by 4.2%/2.1% on dev/test | Helped |
| dense graph in TSG instead of sparse (ablation) | SignGraph TSG · PHOENIX14T dev | WER: WER rises by 1.8% | Hurt |
| sparse graph in LSG instead of dense (ablation) | SignGraph LSG · PHOENIX14T dev | WER: WER rises by 0.7% | Hurt |
| swap CvT/Swin/PyVIG backbones in place of SignGraph backbone (ablation) | CSLR backbone · PHOENIX14T dev | WER: CvT 39.2, Swin 45.4, PyVIG 35.4 vs 17.9 ours | Hurt |
| adopt EdgeConv instead of GCNConv/GATv2/SAGE layer (ablation) | SignGraph GCN layer · PHOENIX14T dev | WER: 17.8 vs 17.9 (GCNConv) | Helped |
| add DropEdge to local dense graph (ablation) | SignGraph LSG · PHOENIX14T dev | WER: 19.2 at 15%, 21.3 at 30% vs 17.8 with none | Hurt |
| adopt Euclidean vs Cosine or Chebyshev distance (ablation) | SignGraph KNN · PHOENIX14T dev | WER: 18.0 cosine, 18.2 Chebyshev, 17.9 Euclidean | No clear effect |
| multiscale stages 8→16→32 or 4→8→16→32 vs 16→32 (ablation) | SignGraph multiscale · PHOENIX14T dev | WER: 18.4 and 18.1 vs 17.8 | Hurt |
| multiscale MultiSignGraph vs single-stage SignGraph | SignGraph · PHOENIX14T test | WER: 20.0 to 19.1 test WER | Helped |
Surprising findings
- Adding DropEdge to the dense local graph degraded performance rather than improving it.
- Adding more LSG and TSG stages does not always help; the two-stage configuration was chosen.
Limitations
- Results on PHOENIX14T test set (19.1%) are only comparable to, not clearly better than, the SOTA on most metrics; the single-model SignGraph is weaker than TwoStream on dev.
- Sparse temporal graph still connects some background nodes, and the hyperparameters K were tuned by grid-like search on the dev set.
- Evaluated only on CSLR (gloss recognition); translation performance is not evaluated despite SLT being mentioned.
Experimental setup
Datasets: PHOENIX14 PHOENIX14T CSL-Daily
Benchmarks: PHOENIX14 PHOENIX14T CSL-Daily
Metrics: word error rate deletion/insertion rate
Baselines: CvT Swin Transformer PyVIG TwoStream CorrNet CoSign HST-GNN VAC SMKD Contrastive STMC C2SLR
Builds on: Vision GNN (VIG) Swin Transformer EdgeConv VAC CTC
Closest papers by meaning
- MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes (NeurIPS 2025 · similarity 0.95)
- TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions (AAAI 2024 · similarity 0.88)
- VSNet: Focusing on the Linguistic Characteristics of Sign Language (CVPR 2025 · similarity 0.88)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.86)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.86)
- Neural Sign Actors: A Diffusion Model for 3D Sign Language Production from Text (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.