Research map / Label-efficient and robust learning / Recognition and long-tail
Dr. Tongue: Sign-Oriented Multi-label Detection for Remote Tongue Diagnosis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Sign-Oriented multi-label framework (ATFE standardization plus SignNet multi-branch attention network) and a telemedicine tongue-image dataset (TongueDx) for detecting eight tongue attributes.
Problem
Recognizing multiple tongue surface attributes from remotely captured, variable-quality tongue images for telehealth tongue diagnosis.
Approach
An adaptive feature extraction pipeline (GroundingDINO/YOLO-MobileNet detection, MobileSAM segmentation, upright orientation) followed by SignNet, which splits the tongue into whole, body and edge branches and fuses color and fur signs via attention, guided by a hand-built attribute-sign relationship table.
Key results
- SignNet reaches average F1 72.87% and Jaccard 75.83% versus 68.80% F1 and 74.21% Jaccard for ResNet50 on five-fold cross-validation.
- ResNet50+ATFE improves average F1 to 70.92%, and ViT and TransFG collapse on Pale and Ecchymosis (F1 near 0).
- SignNet's average accuracy 86.68% is close to practitioners' 85.22% and above laypersons' 49.54%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SignNet instead of ResNet50 | Tongue attribute recognition · TongueDx | Average F1: 68.80±0.48 -> 72.87±0.63 | Helped |
| Add tongue detection (ablation) | ResNet50 · TongueDx | Average F1: 68.80±0.48 -> 69.94±0.60 | Helped |
| Add tongue segmentation (ablation) | ResNet50 · TongueDx | Average F1: 68.80±0.48 -> 69.80±0.51 | Helped |
| Add upright orientation (ablation) | ResNet50 · TongueDx | Average F1: 68.80±0.48 -> 70.92±0.80 | Helped |
| Add the color branch (ablation) | SignNet · TongueDx | Average F1: 71.33±0.38 -> 72.87±0.63 | Helped |
| Add the color branch (ablation) | SignNet · TongueDx | Jaccard: 74.62±0.60 -> 75.83±0.37 | Helped |
| Add the fur branch (ablation) | SignNet · TongueDx | Average F1: 71.61±0.64 -> 72.87±0.63 | Helped |
| Add the fur branch (ablation) | SignNet · TongueDx | Jaccard: 74.74±0.80 -> 75.83±0.37 | Helped |
Surprising findings
- ViT and TransFG obtain near-zero F1 on Pale and Ecchymosis despite reasonable accuracy, reflecting majority-class collapse on imbalanced attributes.
- Practitioners' average accuracy (85.22%) is only modestly higher than the model's, and laypersons perform near chance (~50%).
Limitations
- Severe class imbalance for Pale and Ecchymosis leads to high variance and low F1.
- Performance degrades under extreme lighting; no color correction is applied since the dataset is retrospective.
- Static single images are used, whereas practitioners in real remote diagnosis observe video.
Experimental setup
Datasets: TongueDx
Benchmarks: TongueDx
Metrics: accuracy F1-score Jaccard index
Baselines: ResNet50 ViT TransFG Laypersons Practitioners
Builds on: GroundingDINO Segment Anything (SAM) MobileSAM TransFG
Closest papers by meaning
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.79)
- MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset (NeurIPS 2024 · similarity 0.78)
- DECT: Harnessing LLM-assisted Fine-Grained Linguistic Knowledge and Label-Switched and Label-Preserved Data Generation for Diagnosis of Alzheimer’s Disease (AAAI 2025 · similarity 0.78)
- SCOPE: Sign Language Contextual Processing with Embedding from LLMs (AAAI 2025 · similarity 0.78)
- VSNet: Focusing on the Linguistic Characteristics of Sign Language (CVPR 2025 · similarity 0.78)
- PairAug: What Can Augmented Image-Text Pairs Do for Radiology? (CVPR 2024 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.