doublecringe123/dialoguesum-npc-dialoguesum-stemmed-augmented

---
dataset_info:
features:
- name: inp
dtype: string
- name: target
dtype: string
splits:
- name: train
num_bytes: 47766276
num_examples: 59070
- name: validation
num_bytes: 2437693
num_examples: 3000
- name: test
num_bytes: 5709383
num_examples: 7000
download_size: 26987968
dataset_size: 55913352
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
- split: test
path: data/test-*
---

# Dialoguesum-NPC-dialog-Stemmed-Augmented Dataset

## Overview
This dataset, named "Dialoguesum-Booksum-Stemmed-Augmented," is a custom text summarization dataset created by combining the "kmfoda/booksum" and "knkarthick/dialogsum" datasets. The goal of this dataset is to provide a resource for training and evaluating text summarization models specifically tailored for dialogues and book summaries.

## Dataset Composition
- **Original Sources**:
- [kmfoda/npl-dialog](https://huggingface.co/datasets/npc-engine/light-batch-summarize-dialogue)
- [knkarthick/dialogsum](https://huggingface.co/datasets/knkarthick/dialogsum)
- **Combination Method**: The datasets were concatenated to form a unified corpus.
- **Preprocessing Steps**:
- Stop Word Removal: Common stop words were removed to focus on meaningful content.
- Stemming: Words were stemmed to their base forms to reduce variation.
- Synonym Replacement: Synonyms were replaced to enhance variety in the dataset.

## Data Format
- **Input Format**: Each input instance consists of dialogues and npc-dialogues summaries.
- **Output Format**: Corresponding summary for each input instance.

## Example Usage
```python
from datasets import load_dataset

# Load the custom dataset
dataset = load_dataset("doublecringe123/dialoguesum-npc-dialoguesum-stemmed-augmented")

# Access the training split
train_data = dataset["train"]

# Sample input-output pair
sample = train_data[0]
input_text = sample["inp"]
output_summary = sample["target"]
```

I also recommend to use datasets 2.18.0 version
```
pip install -q datasets>=2.18.0
```

原始信息汇总

Dialoguesum-NPC-dialog-Stemmed-Augmented Dataset

Dataset Composition

• Features:

  • inp: string
  • target: string

• Splits:

  • train: 59070 examples, 47766276 bytes
  • validation: 3000 examples, 2437693 bytes
  • test: 7000 examples, 5709383 bytes

• Download Size: 26987968 bytes

• Dataset Size: 55913352 bytes

Data Files

• Config: default
  • train: data/train-*
  • validation: data/validation-*
  • test: data/test-*