Research map / Video / Video segmentation and tracking
TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCNet, a hybrid CNN-attention network for continuous sign language recognition that combines a trajectory module (self-attention along optical-flow-traced region trajectories) and a correlation module (dynamic sparse attention with learned binary gates), reaching state-of-the-art WER on PHOENIX14, PHOENIX14-T, CSL and CSL-Daily.
Problem
Continuous sign language recognition needs to capture long-range spatio-temporal interactions in video efficiently, while the informative face and hand regions move dynamically and full self-attention is computationally expensive.
Approach
Trace region trajectories with a light-weight optical flow (SPyNet) into a location map so self-attention runs along trajectories of visual tokens, and filter key-value tokens per query with a dynamically gated sparse attention (Improved SemHash binary gates) within each frame.
Key results
- TCNet reaches 18.1/18.9 WER on PHOENIX14 DEV/TEST, versus 18.8/19.4 for CorrNet and 20.5/20.4 for C2SLR (with additional inputs).
- On PHOENIX14-T TCNet achieves 18.3/19.4 WER (DEV/TEST), improving over C2SLR's 20.2/20.4.
- On CSL-Daily TCNet achieves 29.7/29.3 WER versus CorrNet's 30.6/30.1, with lower FLOPs (932.1G) and parameters (19.3M) than compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the trajectory module (ablation) | TCNet baseline without trajectory or correlation · PHOENIX14 | WER: 20.2 -> 19.2; 21.0 -> 19.4 | Helped |
| Add the correlation module (ablation) | TCNet baseline without trajectory or correlation · PHOENIX14 | WER: 20.2 -> 18.8; 21.0 -> 19.8 | Helped |
| Use both trajectory and correlation modules (ablation) | TCNet baseline without trajectory or correlation · PHOENIX14 | WER: 20.2 -> 18.1; 21.0 -> 18.9 | Helped |
| Replace element-wise multiplication with concatenation (ablation) | TCNet · PHOENIX14 | WER: 18.1 -> 18.5; 18.9 -> 19.2 | Hurt |
| Replace element-wise multiplication with summation (ablation) | TCNet · PHOENIX14 | WER: 18.1 -> 18.3; 18.9 -> 19.0 | Hurt |
| Set trajectory subsequence length to 12 (ablation) | TCNet trajectory module · PHOENIX14 | WER: 18.8 -> 18.1 DEV; 19.5 -> 18.9 TEST | Helped |
| Use TCNet blocks with alternative backbones (ablation) | SqueezeNet, ShuffleNet V2, GoogLeNet, and VGG11 · PHOENIX14 | WER: reduction in WER in the range 1.8-2.6% | Helped |
| Use TCNet instead of CorrNet | ResNet-18 CSLR models · PHOENIX14 | WER: 18.8 -> 18.1 DEV; 19.4 -> 18.9 TEST | Helped |
| Use TCNet instead of CorrNet | ResNet-18 CSLR models · PHOENIX14-T | WER: 18.9 -> 18.3 DEV; 20.5 -> 19.4 TEST | Helped |
| Use TCNet instead of CorrNet | CSLR models · CSL | WER: 0.8 -> 0.7 | Helped |
| Use TCNet instead of CorrNet | CSLR models · CSL-Daily TEST | WER: 30.1 -> 29.3 | Helped |
Surprising findings
- Trajectory and correlation modules each reduce WER by similar amounts (about 1-1.6%), and their combination is only partly complementary.
- Element-wise multiplication to combine modules differs from summation or concatenation by only 0.1-0.4% WER.
Limitations
- Performance gain from combining modules is partly redundant, not fully additive.
- Relies on a pre-trained optical flow estimator (SPyNet) and relative frame-subsequence length N chosen by tuning (N=12).
Experimental setup
Datasets: PHOENIX14 PHOENIX14-T CSL CSL-Daily
Benchmarks: PHOENIX14 PHOENIX14-T CSL CSL-Daily
Metrics: WER FLOPs parameters deletions insertions
Baselines: CorrNet SEN C2SLR TLP VAC SMKD FCN SFL STMC DNF
Builds on: CorrNet TLP VAC Improved SemHash SPyNet
Closest papers by meaning
- Continuous Sign Language Recognition With Correlation Network (CVPR 2023 · similarity 0.92)
- SignGraph: A Sign Sequence is Worth Graphs of Nodes (CVPR 2024 · similarity 0.88)
- VSNet: Focusing on the Linguistic Characteristics of Sign Language (CVPR 2025 · similarity 0.87)
- OLMD: Orientation-aware Long-term Motion Decoupling for Continuous Sign Language Recognition (AAAI 2025 · similarity 0.86)
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.86)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.