Finnish-NLP/ai2arc-deepl-translated-sft

---
dataset_info:
features:
- name: instruction
dtype: string
- name: response
dtype: string
splits:
- name: train
num_bytes: 131141
num_examples: 410
download_size: 78634
dataset_size: 131141
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
license: cc-by-sa-4.0
task_categories:
- text-generation
language:
- fi
tags:
- SFT
---

# Dataset Card for Finnish-NLP/ai2arc-deepl-translated-sft

## Creation process
- Load data from allenai/ai2_arc translated with deepl
- Do zero shot classification with facebook/bart-large-mnli with the following prompt:
```python
preds = pipe(f'{row["input"]} is a question about:', candidate_labels=["USA related question", "Math related question", "General question", "Coding related question"])
```
- Filter out rows with too high scores in following categories ["USA related question", "Math related question","Coding related question"]
- Write rows to .txt file with *** on a newline separating instruction/response and then END on a newline separating samples
- Upload file to deepl.com for file translation --> parse samples back from translated files --> Maybe some additional cleaning/filtering based on fasttext langdetect / kenlm perplexity

原始信息汇总

数据集卡片 for Finnish-NLP/ai2arc-deepl-translated-sft

数据集信息

• 特征:
  • instruction: 类型为字符串
  • response: 类型为字符串
• 分割:
  • train: 字节数为131141，样本数为410
• 下载大小: 78634字节
• 数据集大小: 131141字节
• 配置:
  • default: 数据文件路径为data/train-*
• 许可证: cc-by-sa-4.0
• 任务类别:
  • 文本生成
• 语言:
  • 芬兰语
• 标签:
  • SFT

创建过程

1. 从allenai/ai2_arc加载通过DeepL翻译的数据。

2. 使用facebook/bart-large-mnli进行零样本分类，使用以下提示： python preds = pipe(f{row["input"]} is a question about:, candidate_labels=["USA related question", "Math related question", "General question", "Coding related question"])

3. 过滤掉在以下类别中得分过高的行：["USA related question", "Math related question", "Coding related question"]。

4. 将行写入.txt文件，使用***在新行分隔指令/响应，然后使用END在新行分隔样本。

5. 将文件上传到deepl.com进行文件翻译，从翻译后的文件中解析样本，可能基于fasttext langdetect或kenlm perplexity进行额外的清理/过滤。