Research map / Large language models / Language, safety and interpretability

FSboard: Over 3 Million Characters of ASL Fingerspelling Collected via Smartphones

CVPR 2025 · Large language models › Language, safety and interpretability · approach: Sign language datasets and translation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FSboard, a >3M-character ASL fingerspelling dataset collected via Pixel 4A selfie cameras from 147 Deaf signers for smartphone text entry, with a ByT5-Small baseline on MediaPipe Holistic landmarks.

Problem

Machine recognition of sign languages is held back by limited data; fingerspelling recognition for ASL smartphone text entry lacks a large, realistic dataset (prior ones are small, web-scraped, or isolated-image based).

Approach

Collect a large, domain-specific fingerspelling dataset (MacKenzie phrases, URLs, addresses, phone numbers, names) from paid Deaf signers recording one-handed fingerspelling on smartphone selfie cameras, and baseline it by feeding MediaPipe Holistic landmark sequences into a pretrained ByT5-Small encoder-decoder.

Key results

• Best baseline (30 Hz MediaPipe Holistic into ByT5-Small) reaches 11.1% CER and 52.9% top-1 accuracy on the FSboard test set.
• Human raters measured 2.2% CER (74% top-1) with unlimited replay, versus 22.4% CER in a single real-time pass.
• Removing face landmarks raises CER only from 11.1% to 12.0%; 15 Hz frame rate gives 11.8% CER.

Measured effects reported in the paper

Surprising findings

• Training from scratch yields 33.8% CER versus 11.1% with ByT5 pretraining, and the larger ByT5 Base overfits (13.3% CER).
• On a 100-sequence subset the best model (13.5% CER) beats a human's single real-time pass (22.4% CER), though it trails humans with replay.

Limitations

• Phrases are synthetically generated from narrow grammars, so the test set may not reflect real queries.
• Capitalization, special characters and punctuation are underexplored; clip boundaries still contain some issues.
• Only one-handed ASL fingerspelling for a keyboard use case; not sign language translation.

Experimental setup

Datasets: FSboard YouTube captioned videos (bootstrapping) MacKenzie phrase set US Census TIGER 2019 street names April 2022 web crawl URL parts

Benchmarks: FSboard test set FSboard validation set

Metrics: character error rate top-1 accuracy

Baselines: ByT5 Base ChicagoFSWild+ baselines Kaggle competition top score Human raters

Builds on: ByT5 MediaPipe Holistic YouTube-ASL baselines Record These Hands

Closest papers by meaning

• A Wireframe-Based Approach for Classifying and Acquiring Proficiency in the American Sign Language (Student Abstract) (AAAI 2024 · similarity 0.87)
• KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.86)
• MM-WLAuslan: Multi-View Multi-Modal Word-Level Australian Sign Language Recognition Dataset (NeurIPS 2024 · similarity 0.86)
• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.85)
• VSNet: Focusing on the Linguistic Characteristics of Sign Language (CVPR 2025 · similarity 0.85)
• LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition (AAAI 2024 · similarity 0.85)