SuryaKrishna02/aya-telugu-paraphrase

---
annotations_creators:
- expert-generated
language:
- te
language_creators:
- expert-generated
license:
- apache-2.0
multilinguality:
- monolingual
pretty_name: Telugu Paraphrase
size_categories:
- n<1K
source_datasets:
- extended|ai4bharat/IndicXParaphrase
tags:
- paraphrase
task_categories:
- text-generation
task_ids:
- language-modeling
---

# Summary
`aya-telugu-paraphrase` is an open source dataset of instruct-style records generated from the Telugu split of [ai4bharat/IndicXParaphrase](https://huggingface.co/datasets/ai4bharat/IndicXParaphrase/viewer/te/test) dataset. This was created as part of [Aya Open Science Initiative](https://sites.google.com/cohere.com/aya-en/home) from Cohere For AI.

This dataset can be used for any purpose, whether academic or commercial, under the terms of the [Apache 2.0](https://opensource.org/license/apache-2-0) License.

Supported Tasks:
- Training LLMs
- Synthetic Data Generation
- Data Augmentation

Languages: Telugu Version: 1.0

# Dataset Overview
`aya-telugu-paraphrase` is a corpus of more than 1.5k records generated by conversion of Telugu split of [ai4bharat/IndicXParaphrase](https://huggingface.co/datasets/ai4bharat/IndicXParaphrase/viewer/te/test) dataset into Instruct-Style format. This Dataset can be used for the following task:
- Given a sentence, generate a sentence with similar meaning.

# Intended Uses
While immediately valuable for instruction fine tuning large language models, as a corpus of instruction prompts, this dataset also presents a valuable opportunity for synthetic data generation in the methods. For example, prompt-completions could be submitted as few-shot examples to a large open language model to generate sentence and corresponding paraphrased sentence.

# Dataset

## Load with Datasets
To load this dataset with Datasets, you'll just need to install Datasets as `pip install datasets --upgrade` and then use the following code:

```python
from datasets import load_dataset
ds = load_dataset('SuryaKrishna02/aya-telugu-paraphrase')
```

## Purpose of Collection
Telugu is a low-resource language where there no paraphase generation instruct-style dataset to the best of my knowledge. This was created as a part of [Aya Open Science Initiative](https://sites.google.com/cohere.com/aya-en/home) from Cohere For AI to make sure Telugu is well represented in the space of AI/ML. Unlike other datasets that are limited to non-commercial use, this dataset can be used, modified, and extended for any purpose, including academic or commercial applications.

## Sources
- **[ai4bharat/IndicXParaphrase](https://huggingface.co/datasets/ai4bharat/IndicXParaphrase/viewer/te/test)**: Converted this dataset into Instruct-style prompts and completions.

## Data Fields
- `inputs` : Prompt or input to the language model.
- `targets` : Completion or output of the language model.
- `template_id` : Id of the template used in `inputs` and `targets`.
- `template_lang`: ISO code of the language used in the `inputs` and `targets` where *tel* refers to Telugu.

## Templates
For the creation of instruct-style prompts and completions from the original dataset, the following one template category with 6 different variations were used:
1. Given a sentence, generate a sentence with similar meaning.

| template_id | inputs | targets |
|-------------|--------|---------|
| 1 | ```ఈ క్రింది వాక్యం మరోరీతిలో రాయి:\n{{Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |
| 2 | ```ఈ వాక్యం మరోరీతిలో రాయి: {Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |
| 3 | ```ఈ క్రింది వాక్యం ఇంకొలాగా రాయి:\n{{Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |
| 4 | ```ఈ వాక్యం ఇంకొలాగా రాయి: {{Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |
| 5 | ```ఈ క్రింది వాక్యం మరోరకంగా రాయి:\n{{Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |
| 6 | ```ఈ వాక్యం మరోరకంగా రాయి: {{Original Sentence}}``` | ```{{Paraphrased Sentence}}``` |

## Personal or Sensitive Data
This dataset contains public information. To our knowledge, there are no private person’s personal identifiers or sensitive information.

## Language
Telugu

# Known Limitations
- The Dataset is converted from the existing dataset and the contents of this dataset may reflect the bias, factual errors and sensitive matters.
- Although there is utmost care taken to keep the dataset as monolingual, there might be some records that may contain English Language along with Telugu.

# Contributors
[SuryaKrishna02](https://github.com/SuryaKrishna02) and [Desik98](https://github.com/desik1998)

原始信息汇总

数据集概述

基本信息

• 数据集名称: aya-telugu-paraphrase
• 语言: 泰卢固语 (Telugu)
• 数据集大小: 少于1千条记录
• 许可证: Apache 2.0
• 多语言性: 单语种
• 标签: 释义
• 任务类别: 文本生成
• 任务ID: 语言建模

数据集来源

• 源数据集: ai4bharat/IndicXParaphrase
• 创建者: 专家生成

数据集用途

• 支持任务:
  • 训练大型语言模型 (LLMs)
  • 合成数据生成
  • 数据增强
• 具体任务: 给定一个句子，生成一个意义相似的句子。

数据集结构

• 数据字段:
  • inputs: 语言模型的提示或输入
  • targets: 语言模型的完成或输出
  • template_id: 用于inputs和targets的模板ID
  • template_lang: inputs和targets中使用的语言的ISO代码，其中tel指泰卢固语

模板

• 模板类别: 给定一个句子，生成一个意义相似的句子
  • 模板ID: 1-6
  • 示例:
    • inputs: ఈ క్రింది వాక్యం మరోరీతిలో రాయి: {{Original Sentence}}
    • targets: {{Paraphrased Sentence}}

数据集收集目的

• 目的: 泰卢固语是一种低资源语言，目前没有释义生成的指令风格数据集。该数据集作为Cohere For AI的Aya开放科学计划的一部分创建，以确保泰卢固语在AI/ML领域得到充分代表。

已知限制

• 限制:
  • 数据集可能反映源数据集的偏见、事实错误和敏感问题。
  • 尽管尽最大努力保持数据集为单语种，但可能存在一些记录包含泰卢固语和英语混合的情况。

贡献者

• 贡献者: SuryaKrishna02 和 Desik98