Research map / Large language models / Language, safety and interpretability

VAANI: Capturing the language landscape for an inclusive digital India

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Sign language datasets and translation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents VAANI, a large-scale image-prompted, district-stratified Indic speech dataset (31,255 h audio, 2,043 h transcribed, 105 languages, 165 districts) with quality control, plus preliminary ASR and image-retrieval experiments.

Problem

Existing Indic speech datasets cover few languages (mostly the 22 scheduled ones), are language-anchored rather than region-anchored, and are speech-text only, leaving underrepresented languages and regional variation without resources.

Approach

Collect spontaneous speech elicited by district-specific image prompts across 165 districts, pairing each audio with its visual prompt to yield image-speech-text triplets, with a multi-stage automated and manual quality control pipeline.

Key results

• Fine-tuning Whisper-small on VAANI per language (Hindi 331 h, Bengali 101 h, Kannada 80.2 h, Telugu 69.0 h) yields consistent WER improvements over the pre-trained baseline across benchmarks.
• Region-specific Hindi Whisper-small models perform better on geographically proximate states than on distant ones.
• SigLIP2 fine-tuned on Hindi VAANI image-transcription pairs (~45K training images, ~9.5K test images) improves over the original model on image retrieval.

Surprising findings

• Non-finetuned baselines often produce wrong scripts or repetitive hallucinations on low-resource Indic languages, causing WER above 1.0 in some cases.
• Models trained on one state's Hindi data transfer better to nearby states than distant ones, even within the same language.

Limitations

• Transcription and audio coverage is small for many of the 105 languages and concentrated in majority languages.
• Covers 165 of roughly 800 districts, leaving many dialects and minority languages unrepresented.
• Dual-use risk of voice cloning or impersonation from speaker-diverse audio.

Experimental setup

Datasets: VAANI

Benchmarks: GramVAANI FLEURS MUCS CommonVoice Kathbath Kathbath-Noisy VAANI RESPIN

Metrics: word error rate retrieval accuracy

Baselines: Whisper-small (non-finetuned) SigLIP2 (original)

Builds on: Whisper SigLIP2

Closest papers by meaning

• IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS (NeurIPS 2024 · similarity 0.87)
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.86)
• WAXAL: A Large-Scale Multilingual African Language Speech Corpus (NeurIPS 2026 · similarity 0.85)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.84)
• Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.84)
• SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.84)