min-wikisplit-pp

# MinWikiSplit++ This dataset is the HuggingFace version of MinWikiSplit++. MinWikiSplit++ enhances the original [MinWikiSplit](https://aclanthology.org/W19-8615/) by applying two techniques: filtering through NLI classification and sentence-order reversing, which help to remove noise and reduce hallucinations compared to the original MinWikiSplit. The preprocessed MinWikiSplit dataset that formed the basis for this can be found [here](https://huggingface.co/datasets/cl-nagoya/min-wikisplit). ## Dataset Description - **Repository:** https://github.com/nttcslab-nlp/wikisplit-pp - **Paper:** https://arxiv.org/abs/2404.09002 - **Point of Contact:** [Hayato Tsukagoshi](mailto:[e-mail]) ## Usage ```python import datasets as ds dataset: ds.DatasetDict = ds.load_dataset("cl-nagoya/min-wikisplit-pp") print(dataset) # DatasetDict({ # train: Dataset({ # features: ['id', 'complex', 'simple_reversed', 'simple_tokenized', 'simple_original', 'entailment_prob'], # num_rows: 139241 # }) # validation: Dataset({ # features: ['id', 'complex', 'simple_reversed', 'simple_tokenized', 'simple_original', 'entailment_prob'], # num_rows: 17424 # }) # test: Dataset({ # features: ['id', 'complex', 'simple_reversed', 'simple_tokenized', 'simple_original', 'entailment_prob'], # num_rows: 17412 # }) # }) ``` ### Data Fields - id: The ID of the data (note that it is not compatible with the existing MinWikiSplit) - complex: A complex sentence - simple_reversed: Simple sentences with their order reversed - simple_tokenized: A list of simple sentences split by [PySBD](https://github.com/nipunsadvilkar/pySBD), not reversed in order - simple_original: Simple sentences in their original order - entailment_prob: The average probability that each simple sentence is classified as an entailment according to the complex sentence. [DeBERTa-xxl](https://huggingface.co/microsoft/deberta-v2-xxlarge-mnli) is used for the NLI classification. ## Paper Tsukagoshi et al., [WikiSplit++: Easy Data Refinement for Split and Rephrase](https://arxiv.org/abs/2404.09002), LREC-COLING 2024. ## License MinWikiSplit is build upon the [WikiSplit](https://github.com/google-research-datasets/wiki-split) dataset, which is distributed under the CC-BY-SA 4.0 license. Therefore, this dataset follows suit and is distributed under the CC-BY-SA 4.0 license.