BanglaDual: A 1.24-Million Pair Multi-Source Dataset for Banglish Transliteration and Translation

This dataset provides a large-scale, unified resource for Banglish (Romanized Bengali) natural language processing. It contains 1,122,189 Banglish-to-Bangla transliteration pairs and 119,727 Banglish-to-English translation pairs, totaling over 1.24 million aligned texts. The data is aggregated from six distinct public sources for transliteration and four for translation, encompassing diverse domains such as general Wikipedia text (85.34%), emotion detection (7.12%), hate speech (3.33%), academic transliteration benchmarks, and regional dialects. A key feature of this dataset is its unified schema, which resolves the fragmented nature of previous Banglish resources. All textual data underwent rigorous standardization, including Unicode NFKC normalization to handle Bengali character variations, cross-source deduplication, and the removal of missing values. Furthermore, every entry is tagged with its original source, allowing researchers to filter data by domain. To support immediate model development, the dataset includes a pre-computed 80/10/10 train/validation/test split (seed = 42). This dataset is roughly 30 times larger than previously available benchmarks like BanglaTLit and is the first to support both transliteration and translation tasks from a single format. It aims to support researchers addressing the challenges of code-mixed Bengali text in low-resource environments. Repository File Structure:To facilitate ease of use, the repository is organized into five directories containing the data files in Apache Parquet format (with Snappy compression for efficiency) and standard CSVs for quick inspection. 📁 Task-Specific-Datasets/banglish_bangla_transliteration.parquet: Contains 1,122,189 cleaned and deduplicated transliteration pairs. banglish_english_translation.parquet: Contains 119,727 translation pairs, including 470 manually translated by the authors. 📁 Merged-Dataset/banglish_merged_dual_task.parquet: Contains 1,122,998 rows produced by an outer join of the Bangla and English datasets, supporting multi-task learning. 📁 Model-Splits/train.parquet: Training set partition (898,024 rows). validation.parquet: Validation set partition (112,253 rows). test.parquet: Testing set partition (112,254 rows). 📁 Sample-Data/samples_bangla.csv: 60 randomly sampled transliteration records for quick inspection. samples_english.csv: 40 randomly sampled translation records for quick inspection. sample_merged_data.csv: A representative sample showcasing the outer-joined dual-task format structure. 📁 Processing-Scripts/banglish_to_bangla_pipeline.py: Code to process and merge the six Bangla sources. banglish_to_english_pipeline.py: Code to process the four English sources. merge_datasets.py: Code to perform the outer join and generate the 80/10/10 splits.