mpasila/ParallelFiction-Ja_En-100k-alpaca-4k-context

---
license: apache-2.0
task_categories:
- translation
language:
- ja
- en
---
This is a modified version of [NilanE/ParallelFiction-Ja_En-100k](https://huggingface.co/datasets/NilanE/ParallelFiction-Ja_En-100k) which has been turned into Alpaca format.

This has also been chunked for 4096 tokens for [augmxnt/shisa-base-7b-v1](https://huggingface.co/augmxnt/shisa-base-7b-v1) model's tokenizer.

If you want the non chunked version it's [here](https://huggingface.co/datasets/mpasila/ParallelFiction-Ja_En-100k-alpaca).

# Dataset format (correct one)
```json
{
'instruction' : 'Japanese chapter'
'output' : 'English translation'
'input' : 'empty'
}
```

# Original Dataset card

# Dataset details
Each entry in this dataset is a sentence-aligned Japanese web novel chapter and English fan translation.
The intended use-case is for document translation tasks.

# Dataset format
```json
{
'src' : 'JAPANESE CHAPTER'
'trg' : 'ENGLISH TRANSLATION'
'meta' : {
"source": 'SAME ACROSS ALL ENTRIES',
"series": 'NAME OF WEB NOVEL SERIES',
"missed_lines": 'NUMBER OF LINES THAT WERE AT THE SAME INDEX BUT NOT DETECTED AS BEING TRANSLATIONS OF EACH OTHER',
"inserted_lines_src": 'NUMBER OF LINES IN THE JAPANESE TEXT THAT DID NOT HAVE A MATCHING TRANSLATION BUT ARE BUFFERED BY TRANSLATED LINES',
"inserted_lines_trg": 'SAME AS ABOVE BUT FOR ENGLISH',
}
}
```
A high number of inserted lines is not necessarily a sign of a bad pair, as many translations concatenate or divide source chapters when publishing.
Instead, watch out for high numbers of missed lines or entries where the inserted line count is high for both source and target.

原始信息汇总

数据集概述

数据集名称

• 修改自 NilanE/ParallelFiction-Ja_En-100k，转换为Alpaca格式。

许可证

• Apache-2.0

任务类别

• 翻译

语言

• 日语 (ja)
• 英语 (en)

数据集格式

• 每个条目包含日语网络小说章节和对应的英语粉丝翻译。
• 格式示例： json { src : JAPANESE CHAPTER, trg : ENGLISH TRANSLATION, meta : { "source": SAME ACROSS ALL ENTRIES, "series": NAME OF WEB NOVEL SERIES, "missed_lines": NUMBER OF LINES THAT WERE AT THE SAME INDEX BUT NOT DETECTED AS BEING TRANSLATIONS OF EACH OTHER, "inserted_lines_src": NUMBER OF LINES IN THE JAPANESE TEXT THAT DID NOT HAVE A MATCHING TRANSLATION BUT ARE BUFFERED BY TRANSLATED LINES, "inserted_lines_trg": SAME AS ABOVE BUT FOR ENGLISH, } }

数据集用途

• 文档翻译任务

注意事项

• 高数量的插入行不一定表示配对质量差，因为许多翻译在发布时会合并或分割源章节。
• 应关注高数量的遗漏行或源和目标中插入行数量都高的条目。