ShadhuCholito-BN

## Dataset Collection, Translation, and Preparation Process The dataset was created to support Bengali language style classification research, particularly for distinguishing between **Cholito Bangla** (colloquial Bengali) and **Sadhu Bangla** (classical/formal Bengali). Text data were collected from multiple Bengali literary sources, including novels, stories, and digital text-based books written by well-known Bengali authors. Initially, text was extracted from digital Bengali documents and converted into machine-readable format. Sentence segmentation techniques were applied to divide paragraphs into individual sentences using Bengali punctuation marks and newline patterns. Each sentence was then manually or semi-automatically labeled as either `cholito` or `sadhu` according to its linguistic structure and writing style. After extraction and labeling, multiple datasets from different sources were merged into a unified dataset structure containing two columns: `sentence` and `type`. Several preprocessing operations were performed to improve dataset quality, including removal of duplicate entries, elimination of empty rows, normalization of text formatting, and correction of inconsistent labels such as capitalization differences (e.g., “Cholito” vs “cholito”). To create an English version of the dataset, the Bangla sentences were translated into English using the `GoogleTranslator` module from the `deep-translator` library. A batch translation strategy was implemented to improve processing efficiency and reduce API requests by translating multiple sentences simultaneously. A retry mechanism was also incorporated to handle temporary API failures and connection issues during translation. The translated outputs were stored as a parallel English dataset while preserving their original labels. After completing the translation process, additional cleaning operations were performed to remove missing values, empty strings, translation errors, and duplicate entries from both Bangla and English datasets. The final datasets were then shuffled randomly to reduce source-wise ordering bias and improve data distribution. Finally, both the Bangla and English datasets were divided into training and testing subsets using a 70:30 ratio. The training datasets were prepared for machine learning model training, while the testing datasets were reserved for evaluation and performance analysis. All datasets were stored in CSV format using UTF-8 encoding to ensure proper multilingual text representation and compatibility across different NLP and machine learning frameworks.数据集收集、翻译与制备流程 本数据集旨在支撑孟加拉语语体分类研究，尤其针对**Cholito Bangla（口语体孟加拉语）**与**Sadhu Bangla（书面正式体孟加拉语）**的区分任务。文本数据采集自多个孟加拉语文学来源，涵盖知名孟加拉语作家创作的小说、短篇故事及数字化文本书籍。 初始阶段，研究人员从数字化孟加拉语文档中提取文本并转换为机器可读格式。随后采用分句技术，基于孟加拉语标点符号与换行模式，将段落拆解为独立句子。后续研究人员依据句子的语言结构与书写风格，以人工或半自动方式为每条句子标注`cholito`（口语体）或`sadhu`（书面体）标签。 完成提取与标注后，来自不同来源的多份数据集被整合为统一的数据集结构，包含`sentence`（句子）与`type`（语体类型）两列。为提升数据集质量，研究人员执行了多项预处理操作：移除重复条目、删除空行、规范化文本格式，以及修正标签不一致问题（例如"Cholito"与"cholito"的大小写差异）。 为生成该数据集的英文版本，研究人员借助`deep-translator`库中的`GoogleTranslator`模块，将孟加拉语句子译为英文。为提升处理效率并减少API调用次数，研究人员采用批量翻译策略同时处理多条句子；此外还增设重试机制，以应对翻译过程中临时出现的API故障与连接问题。翻译结果以平行英文数据集形式存储，并保留原语体标签。 翻译流程完成后，研究人员开展额外清洗操作，移除孟加拉语与英文数据集中的缺失值、空字符串、翻译错误条目及重复内容。随后对最终数据集进行随机洗牌，以消除来源相关的顺序偏差，优化数据分布。 最后，研究人员将孟加拉语与英文数据集按照70:30的比例划分为训练集与测试集。训练集用于机器学习模型的训练，测试集则留作模型评估与性能分析使用。 所有数据集均采用UTF-8编码的CSV格式存储，以确保多语言文本的正确表示，并兼容各类自然语言处理（Natural Language Processing）与机器学习框架。