UrduAbuse-50K: An Annotated Corpus of Urdu Abusive Language from Twitter

Social-media platforms have become the primary communication channels for Urdu speakers; however, automated content moderation systems that natively process the Nastaliq script remain absent. This gap leaves Urdu-speaking communities with significantly less protection from abusive online content than users of high-resource languages. This study addresses two interconnected challenges: the scarcity of large, high-quality annotated datasets for Urdu abusive language detection and the absence of a systematic comparative evaluation of machine learning and transformer-based approaches for this task. We introduce UrduAbuse-50K, a novel annotated corpus of 50,051 Urdu tweets labelled as abusive or non-abusive, developed through a rigorous six-phase protocol that includes keyword-based data acquisition, systematic filtration, annotation guideline development, manual annotation by three expert annotators, inter-annotator agreement validation (Fleiss Kappa = 0.82), and quality assurance. Using this corpus, we evaluated five classification models: three traditional machine learning classifiers (LinearSVC, Logistic Regression, and Random Forest) and two fine-tuned transformer-based models (BERT Urdu and RoBERTa Urdu). RoBERTa Urdu achieved the highest performance, with an accuracy of 96.13% and an F1-score of 96.02%. McNemar’s test (p < 0.01) statistically confirmed that it outperformed BERT Urdu. An ablation study identified text normalization as the most critical preprocessing step, while an error analysis revealed that sarcasm, code-switching, and context-dependent language collectively accounted for 81.8% of misclassifications. SHAP-based explainability analysis suggests that the model relies on semantically meaningful, token-level features. Together, these contributions address a long-standing resource deficit in Urdu NLP and establish a rigorous evaluation framework for future research.