Research map / Large language models / Language, safety and interpretability
IndicCONAN: A Multilingual Dataset for Combating Hate Speech in Indian Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IndicCONAN, a ~2,500-pair Hindi and English hate speech/counter-narrative dataset built via a human-in-the-loop generate-correct cycle, and trains type-conditioned mGPT generators.
Problem
Counter-narrative research for hate speech is scarce in non-English languages, especially Hindi and the Indian context, where hate speech has caused violence and discrimination.
Approach
A scalable human-in-the-loop pipeline in which an autoregressive mGPT model, trained on annotated pairs, generates counter-narratives that annotators then validate and edit, with each cycle's augmented data used for retraining; generation is conditioned on hate speech, hate category and CN type.
Key results
- English CNs from the best pipeline (M4) reached mean perplexity 3.85 (M3) and toxicity-style Perspective score 0.04, with diversity (Self-BLEU) 0.08.
- Hindi CNs scored perplexity 3.70, Perspective score 0.06 and Self-BLEU 0.14 under M4, the best Hindi pipeline.
- Conditioning on CN type (M2, M4) yields higher relevance metrics than conditioning only on hate speech (M1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CN type conditioning (ablation) | mGPT CN generation (M1 to M2) · English test | BLEU-2: 0.35 -> 0.38 | Helped |
| Add CN type conditioning (ablation) | mGPT CN generation (M1 to M2) · Hindi test | ROUGE-1: 0.22 -> 0.30 | Helped |
| Add hate speech type conditioning (ablation) | mGPT CN generation (M2 to M4) · Hindi test | BERTScore: 0.72 -> 0.75 | Helped |
| Add hate speech type conditioning (ablation) | mGPT CN generation (M2 to M4) · English test | BERTScore: 0.87 -> 0.89 | Helped |
| Add hate speech type conditioning (ablation) | mGPT CN generation (M2 to M4) · English test | BLEU-2: 0.38 -> 0.37 | Hurt |
| Add both hate speech type and CN type conditioning (ablation) | mGPT CN generation (M1 to M4) · Hindi test | ROUGE-1: 0.22 -> 0.32 | Helped |
| Add both hate speech type and CN type conditioning (ablation) | mGPT CN generation (M1 to M4) · English test | Perplexity: 4.87 -> 3.95 | Helped |
Surprising findings
- Diversity scores are low across all pipelines, which the authors attribute to diversity in the initial annotated data rather than to the models.
Limitations
- Only Hindi and Indian English covered; other Indian languages are future work.
- Automatic metrics only; no human evaluation of CN quality reported in the main text.
- Relies on a small seed set (475 training pairs) and a limited set of three annotators.
Experimental setup
Datasets: Hostility Detection dataset (Bhardwaj et al. 2020) HASOC 2019 Hindi subset
Benchmarks: IndicCONAN test set
Metrics: Self-BLEU BLEU-2 BLEU-3 ROUGE-1 ROUGE-L BERTScore Perspective score perplexity
Baselines: M1 M2 M3
Builds on: mGPT CONAN (Chung et al. 2019)
Closest papers by meaning
- Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales (AAAI 2024 · similarity 0.86)
- A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings (NeurIPS 2025 · similarity 0.83)
- IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS (NeurIPS 2024 · similarity 0.82)
- BeyondGender: A Multifaceted Bilingual Dataset for Practical Sexism Detection (AAAI 2025 · similarity 0.81)
- Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media (AAAI 2024 · similarity 0.81)
- VAANI: Capturing the language landscape for an inclusive digital India (NeurIPS 2026 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.