nirmalendu01/samanantar-parallel-en-hi-bn-ta-te

samanantar-parallel-en-hi-bn-ta-te数据集是一个多语言平行语料库，包含英语、印地语、孟加拉语、泰米尔语和泰卢固语的行对齐平行句子。该数据集源自Samanantar v0.3.0（使用CC BY-NC 4.0许可证），通过在原数据集的英语src字段上进行内连接构建，确保跨语言配置（hi、bn、ta、te）中的句子基于剥离后的src字符串精确匹配。数据集包含五列：en（英语原文）、hi（印地语翻译）、bn（孟加拉语翻译）、ta（泰米尔语翻译）和te（泰卢固语翻译），适用于机器翻译等NLP任务。注意：不同语言配置中的idx不共享。

The samanantar-parallel-en-hi-bn-ta-te dataset is a multilingual parallel corpus containing row-aligned parallel sentences for English, Hindi, Bengali, Tamil, and Telugu. It is built from the ai4bharat/samanantar dataset (specifically version 0.3.0, under CC BY-NC 4.0 license) by inner-joining on the English src field, with exact matching on the stripped src string across the language configurations (hi, bn, ta, te). The dataset includes columns: en (English source from Samanantar), hi (Hindi translation), bn (Bengali translation), ta (Tamil translation), and te (Telugu translation). It is designed for tasks such as machine translation, and note that the idx is not shared across language configs.