ChattoBan: A Benchmark Dataset for Language Identification Between Bengali and Chittagonian Dialects

Chittagonian is one of the most widely spoken native languages in Bangladesh, with an estimated 14 million speakers across the country and abroad. Although Bengali is the national language, Chittagonian differs significantly in phonology, vocabulary, and grammar. These linguistic differences make automatic language identification an important task for NLP applications such as machine translation, language detection, and sentiment analysis.

To address the scarcity of Chittagonian language resources, we introduce ChattoBan, a benchmark dataset designed for sentence-level identification between Bengali and Chittagonian. The dataset contains 6,151 annotated sentences, categorized as follows:

Chittagonian: 2,650 sentences

Bengali: 3,501 sentences

Chittagonian sentences were collected from social media platforms (Facebook, Twitter), Chittagonian news articles, song lyrics, and direct contributions from native speakers. Bengali sentences were sourced from various Bengali newspapers and classical literature to ensure authentic and diverse language representation.

To ensure annotation reliability, two native Chittagonian speakers and one native Bengali speaker independently reviewed and validated all sentence labels. Additionally, preprocessing steps such as duplicate removal, punctuation removal, and English character and number filtering were applied to enhance data quality while preserving linguistic authenticity.

The ChattoBan dataset has significant implications across multiple NLP and AI domains, including:

Language identification for closely related languages

Machine translation and code-switching analysis

Supervised and semi-supervised learning

Sociolinguistic and dialect studies

Bangla-centric NLP research and educational applications

The ChattoBan dataset is openly available for academic and research purposes, promoting collaboration and innovation within the Bangla NLP community. By providing a reliable benchmark for Bengali–Chittagonian identification, this dataset aims to support future advancements in low-resource language processing.

吉大港语（Chittagonian）是孟加拉国使用最广泛的本土语言之一，据估算在孟加拉国境内外共有1400万使用者。尽管孟加拉语（Bengali）是孟加拉国的官方语言，但吉大港语在音系、词汇与语法层面均存在显著差异。这类语言差异使得自动语言识别成为机器翻译、语言检测、情感分析等自然语言处理（Natural Language Processing, NLP）应用中的关键任务。

为解决吉大港语语言资源匮乏的问题，我们推出ChattoBan基准数据集，该数据集专为孟加拉语与吉大港语的句子级识别任务设计。数据集共包含6151条标注句子，分类如下：
吉大港语：2650条句子
孟加拉语：3501条句子

吉大港语语料采集自社交媒体平台（Facebook、Twitter）、吉大港语新闻文章、歌词以及母语者的直接投稿。孟加拉语语料则来源于各类孟加拉语报纸与经典文学作品，以确保语言表征的真实性与多样性。

为保证标注可靠性，两名吉大港语母语者与一名孟加拉语母语者独立审核并验证了所有句子的标签。此外，我们还执行了去重、标点移除、英文字符与数字过滤等预处理步骤，在保留语言原生性的同时提升数据质量。

ChattoBan数据集在多个自然语言处理与人工智能领域具备重要应用价值，涵盖：
近缘语言的语言识别任务
机器翻译与语码转换分析
监督学习与半监督学习
社会语言学与方言研究
孟加拉语聚焦型自然语言处理研究与教育应用

ChattoBan数据集面向学术与研究用途公开可用，旨在推动孟加拉语自然语言处理社区的合作与创新。通过为孟加拉语-吉大港语识别任务提供可靠基准，该数据集旨在助力低资源语言处理领域的未来发展。