BPCC

# BPCC Dataset ## Training Bharat Parallel Corpus Collection (BPCC) is a comprehensive and publicly available parallel corpus that includes both existing and new data for all 22 scheduled Indic languages. It is comprised of two parts: BPCC-Mined and BPCC-Human, totaling approximately 230 million bitext pairs. BPCC-Mined contains about 228 million pairs, with nearly 126 million pairs newly added as a part of this work. On the other hand, BPCC-Human consists of 2.2 million gold standard English-Indic pairs, with an additional 644K bitext pairs from English Wikipedia sentences (forming the BPCC-H-Wiki subset) and 139K sentences covering everyday use cases (forming the BPCC-H-Daily subset). It is worth highlighting that BPCC provides the first available datasets for 7 languages and significantly increases the available data for all languages covered. You can find the contribution from different sources in the following table: <table> <tbody> <tr> <td rowspan="4">BPCC-Mined</th> <td rowspan="2">Existing</th> <td>Samanantar</th> <td>19.4M</th> </tr> <tr> <td>NLLB</th> <td>85M</th> </tr> <tr> <td rowspan="2">Newly Added</th> <td>Samanantar++</th> <td>121.6M</th> </tr> <tr> <td>Comparable</th> <td>4.3M</th> </tr> <tr> <td rowspan="5">BPCC-Human</td> <td rowspan="3">Existing</td> <td>NLLB</td> <td>18.5K</td> </tr> <tr> <td>ILCI</td> <td>1.3M</td> </tr> <tr> <td>Massive</td> <td>115K</td> </tr> <tr> <td rowspan="2">Newly Added</td> <td>Wiki</td> <td>644K</td> </tr> <tr> <td>Daily</td> <td>139K</td> </tr> </tbody> </table> Additionally, we provide augmented back-translation data generated by our intermediate IndicTrans2 models for training purposes. Please refer our paper for more details on the selection of sample proportions and sources. <table> <tbody> <tr> <td>English BT data (English Original)</td> <td>401.9M</td> </tr> <tr> <td>Indic BT data (Indic Original)</td> <td>400.9M</td> </tr> </tbody> </table> <br> ## Evaluation IN22 test set is a newly created comprehensive benchmark for evaluating machine translation performance in multi-domain, n-way parallel contexts across 22 Indic languages. It has been created from three distinct subsets, namely IN22-Wiki, IN22-Web and IN22-Conv. The Wikipedia and Web sources subsets offer diverse content spanning news, entertainment, culture, legal, and India-centric topics. IN22-Wiki and IN22-Web have been combined and considered for evaluation purposes and released as IN22-Gen. Meanwhile, IN22-Conv the conversation domain subset is designed to assess translation quality in typical day-to-day conversational-style applications. <table> <tbody> <tr> <td>IN22-Gen (IN22-Wiki + IN22-Web)</td> <td>1024 sentences</td> <td>🤗 <a href="https://huggingface.co/datasets/ai4bharat/IN22-Gen">ai4bharat/IN22-Gen</td> </tr> <tr> <td>IN22-Conv</td> <td>1503 sentences</td> <td>🤗 <a href="https://huggingface.co/datasets/ai4bharat/IN22-Conv">ai4bharat/IN22-Conv</td> </tr> </tbody> </table> ## LICENSE The following table lists the licenses associated with the different artifacts released as a part of this work: | Artifact | LICENSE | | ----------------------------------------------------- | --------------------------------------------------------------------- | | Existing Mined Corpora (NLLB & Samanantar) | [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) | | Existing Seed Corpora (NLLB-Seed, ILCI, MASSIVE) | [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) | | Newly Added Mined Corpora (Samanantar++ & Comparable) | [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) | | Newly Added Seed Corpora (BPCC-H-Wiki & BPCC-H-Daily) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | | Newly Created IN-22 test set (IN22-Gen & IN22-Conv) | [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) | | Back-translation data (BPCC-BT) | [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) | | Model checkpoints | [MIT](https://github.com/ai4bharat/IndicTrans2/blob/main/LICENSE) | The mined corpora collection (BPCC-Mined), existing seed corpora (NLLB-Seed, ILCI, MASSIVE), Backtranslation data (BPCC-BT), are released under the following licensing scheme: - We do not own any of the text from which this data has been extracted. - We license the actual packaging of this data under the Creative Commons [CC0 license (“no rights reserved”)](https://creativecommons.org/share-your-work/public-domain/cc0/). - To the extent possible under law, [AI4Bharat](https://ai4bharat.iitm.ac.in/) has waived all copyright and related or neighboring rights to BPCC-Mined, existing seed corpora (NLLB-Seed, ILCI, MASSIVE) and BPCC-BT. ## Citation ```bibtex @article{gala2023indictrans, title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages}, author={Jay Gala and Pranjal A Chitale and A K Raghavan and Varun Gumma and Sumanth Doddapaneni and Aswanth Kumar M and Janki Atul Nawale and Anupama Sujatha and Ratish Puduppully and Vivek Raghavan and Pratyush Kumar and Mitesh M Khapra and Raj Dabre and Anoop Kunchukuttan}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2023}, url={https://openreview.net/forum?id=vfT4YuzAYA}, note={} } ```