dialogstudio|对话式人工智能数据集|数据集数据集

# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
**Author**: [Jianguo Zhang](https://github.com/jianguoz), [Kun Qian](https://github.com/qbetterk)

[Paper](https://arxiv.org/pdf/2307.10172.pdf)|[Github](https://github.com/salesforce/DialogStudio)|[GDrive]

🎉 **March 18, 2024: Update for AI Agent**. Check [xLAM](https://github.com/SalesforceAIResearch/xLAM) for the latest data and models relevant to AI Agent!

🎉 **March 10 2024: Update for dataset viewer issues:**
- Please refer to https://github.com/salesforce/DialogStudio for view of each dataset, where we provide 5 converted examples along with 5 original examples under each data folder.
- For example, https://github.com/salesforce/DialogStudio/tree/main/open-domain-dialogues/ShareGPT contains two files: [converted_examples.json](https://github.com/salesforce/DialogStudio/blob/main/open-domain-dialogues/ShareGPT/converted_example.json) and [original_example.json](https://github.com/salesforce/DialogStudio/blob/main/open-domain-dialogues/ShareGPT/original_example.json).

**Follow the [DialogStudio](https://github.com/salesforce/DialogStudio) GitHub repository for latest information.**

### Datasets

### Load dataset
The datasets are split into several categories in HuggingFace
```
Datasets/
├── Knowledge-Grounded-Dialogues
├── Natural-Language-Understanding
├── Open-Domain-Dialogues
├── Task-Oriented-Dialogues
├── Dialogue-Summarization
├── Conversational-Recommendation-Dialogs
```

You can load any dataset in the DialogStudio from the [HuggingFace hub](https://huggingface.co/datasets/Salesforce/dialogstudio) by claiming the `{dataset_name}`, which is exactly the dataset folder name. All available datasets are described in [dataset content](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv). For easier reference, [available dataset names](#Available Datasets) are also listed below.

Below is one example to load the [MULTIWOZ2_2](https://huggingface.co/datasets/Salesforce/dialogstudio/blob/main/task_oriented/MULTIWOZ2_2.zip) dataset under the [task-oriented-dialogues](https://huggingface.co/datasets/Salesforce/dialogstudio/tree/main/task_oriented) category:

Load the dataset
```python
from datasets import load_dataset

dataset = load_dataset('Salesforce/dialogstudio', 'MULTIWOZ2_2')
```
Here is the output structure of MultiWOZ 2.2
```python
DatasetDict({
train: Dataset({
features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
num_rows: 8437
})
validation: Dataset({
features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
num_rows: 1000
})
test: Dataset({
features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
num_rows: 1000
})
})
```

### Available Datasets

The ``data_name`` for ``load_dataset("Salesforce/dialogstudio", data_name)`` can be found below. More detailed information for each dataset can be found in out [github](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv).
```python
"natural_language_understanding": [
"ATIS", "ATIS-NER", "BANKING77", "BANKING77-OOS", "CLINC-Single-Domain-OOS-banking",
"CLINC-Single-Domain-OOS-credit_cards", "CLINC150", "DSTC8-SGD", "HWU64", "MIT-Movie",
"MIT-Restaurant", "RESTAURANTS8K", "SNIPS", "SNIPS-NER", "TOP", "TOP-NER"
],
"task_oriented": [
"ABCD", "AirDialogue", "BiTOD", "CaSiNo", "CraigslistBargains",
"Disambiguation", "DSTC2-Clean", "FRAMES", "GECOR", "HDSA-Dialog",
"KETOD", "KVRET", "MetaLWOZ", "MS-DC", "MuDoCo",
"MulDoGO", "MultiWOZ_2.1", "MULTIWOZ2_2", "SGD", "SimJointGEN",
"SimJointMovie", "SimJointRestaurant", "STAR", "Taskmaster1", "Taskmaster2",
"Taskmaster3", "WOZ2_0"
],
"dialogue_summarization": [
"AMI", "CRD3", "DialogSum", "ECTSum", "ICSI",
"MediaSum", "QMSum", "SAMSum", "TweetSumm", "ConvoSumm",
"SummScreen_ForeverDreaming", "SummScreen_TVMegaSite"
],
"conversational_recommendation": [
"Redial", "DuRecDial-2.0", "OpenDialKG", "SalesBot",
],
"open_domain": [
"chitchat-dataset", "ConvAI2", "AntiScam", "Empathetic", "HH-RLHF",
"PLACES3.5", "Prosocial", "SODA", "ShareGPT"
],
"knowledge_grounded": [
"CompWebQ", "CoQA", "CoSQL", "DART", "FeTaQA",
"GrailQA", "HybridQA", "MTOP", "MultiModalQA", "SParC",
"Spider", "SQA", "ToTTo", "WebQSP", "WikiSQL",
"WikiTQ", "wizard_of_internet", "wizard_of_wikipedia"
],
```

# License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio:
- A portion of these datasets is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).
- Some retain their original licenses even after modification.
- For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.

# Ethical Considerations
This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.

# Citation
The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own (Accepted by EACL 2024 Findings as a long paper).

```
@article{zhang2023dialogstudio,
title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI},
author={Zhang, Jianguo and Qian, Kun and Liu, Zhiwei and Heinecke, Shelby and Meng, Rui and Liu, Ye and Yu, Zhou and Savarese, Silvio and Xiong, Caiming},
journal={arXiv preprint arXiv:2307.10172},
year={2023}
}
```