ConvLab/sgd5

---
language:
- en
license:
- cc-by-sa-4.0
multilinguality:
- monolingual
pretty_name: SGD-X v5
size_categories:
- 10K<n<100K
task_categories:
- conversational
---

# Dataset Card for SGD-X v5

- **Repository:** https://github.com/google-research-datasets/dstc8-schema-guided-dialogue/tree/master/sgd_x
- **Paper:** https://arxiv.org/pdf/2110.06800.pdf
- **Leaderboard:** None
- **Who transforms the dataset:** Qi Zhu(zhuq96 at gmail dot com)

To use this dataset, you need to install [ConvLab-3](https://github.com/ConvLab/ConvLab-3) platform first. Then you can load the dataset via:
```
from convlab.util import load_dataset, load_ontology, load_database

dataset = load_dataset('sgd5')
ontology = load_ontology('sgd5')
database = load_database('sgd5')
```
For more usage please refer to [here](https://github.com/ConvLab/ConvLab-3/tree/master/data/unified_datasets).

### Dataset Summary

The **Schema-Guided Dialogue (SGD)** dataset consists of over 20k annotated multi-domain, task-oriented conversations between a human and a virtual assistant. These conversations involve interactions with services and APIs spanning 20 domains, such as banks, events, media, calendar, travel, and weather. For most of these domains, the dataset contains multiple different APIs, many of which have overlapping functionalities but different interfaces, which reflects common real-world scenarios. The wide range of available annotations can be used for intent prediction, slot filling, dialogue state tracking, policy imitation learning, language generation, and user simulation learning, among other tasks for developing large-scale virtual assistants. Additionally, the dataset contains unseen domains and services in the evaluation set to quantify the performance in zero-shot or few-shot settings.

The **SGD-X** dataset consists of 5 linguistic variants of every schema in the original SGD dataset. Linguistic variants were written by hundreds of paid crowd-workers. In the SGD-X directory, v1 represents the variant closest to the original schemas and v5 the farthest in terms of linguistic distance. To evaluate model performance on SGD-X schemas, dialogues must be converted using the script generate_sgdx_dialogues.py.

- **How to get the transformed data from original data:**
- Download [dstc8-schema-guided-dialogue-master.zip](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue/archive/refs/heads/master.zip).
- Modified `sgd_x/generate_sgdx_dialogues.py` as https://github.com/google-research-datasets/dstc8-schema-guided-dialogue/issues/57
- Run `python -m sgd_x.generate_sgdx_dialogues` under `dstc8-schema-guided-dialogue-master` dir which need tensorflow installed.
- Run `python preprocess.py` in the current directory.
- **Main changes of the transformation:**
- Lower case original `act` as `intent`.
- Add `count` slot for each domain, non-categorical, find span by text matching.
- Categorize `dialogue acts` according to the `intent`.
- Concatenate multiple values using `|`.
- Retain `active_intent`, `requested_slots`, `service_call`.
- **Annotations:**
- dialogue acts, state, db_results, service_call, active_intent, requested_slots.

### Supported Tasks and Leaderboards

NLU, DST, Policy, NLG, E2E

### Languages

English

### Data Splits

| split | dialogues | utterances | avg_utt | avg_tokens | avg_domains | cat slot match(state) | cat slot match(goal) | cat slot match(dialogue act) | non-cat slot span(dialogue act) |
|------------|-------------|--------------|-----------|--------------|---------------|-------------------------|------------------------|--------------------------------|-----------------------------------|
| train | 16142 | 329964 | 20.44 | 9.75 | 1.84 | 100 | - | 100 | 100 |
| validation | 2482 | 48726 | 19.63 | 9.66 | 1.84 | 100 | - | 100 | 100 |
| test | 4201 | 84594 | 20.14 | 10.4 | 2.02 | 100 | - | 100 | 100 |
| all | 22825 | 463284 | 20.3 | 9.86 | 1.87 | 100 | - | 100 | 100 |

45 domains: ['Banks_15', 'Buses_15', 'Buses_25', 'Calendar_15', 'Events_15', 'Events_25', 'Flights_15', 'Flights_25', 'Homes_15', 'Hotels_15', 'Hotels_25', 'Hotels_35', 'Media_15', 'Movies_15', 'Music_15', 'Music_25', 'RentalCars_15', 'RentalCars_25', 'Restaurants_15', 'RideSharing_15', 'RideSharing_25', 'Services_15', 'Services_25', 'Services_35', 'Travel_15', 'Weather_15', 'Alarm_15', 'Banks_25', 'Flights_35', 'Hotels_45', 'Media_25', 'Movies_25', 'Restaurants_25', 'Services_45', 'Buses_35', 'Events_35', 'Flights_45', 'Homes_25', 'Media_35', 'Messaging_15', 'Movies_35', 'Music_35', 'Payment_15', 'RentalCars_35', 'Trains_15']
- **cat slot match**: how many values of categorical slots are in the possible values of ontology in percentage.
- **non-cat slot span**: how many values of non-categorical slots have span annotation in percentage.

### Citation

```
@inproceedings{lee2022sgd,
title={SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems},
author={Lee, Harrison and Gupta, Raghav and Rastogi, Abhinav and Cao, Yuan and Zhang, Bin and Wu, Yonghui},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={36},
number={10},
pages={10938--10946},
year={2022}
}
```

### Licensing Information

[**CC BY-SA 4.0**](https://creativecommons.org/licenses/by-sa/4.0/)

原始信息汇总

数据集卡片 for SGD-X v5

数据集概述

Schema-Guided Dialogue (SGD) 数据集包含超过20k个多领域、任务导向的对话，这些对话是人类与虚拟助手之间的交互。这些对话涉及与20个不同领域的服务和API的交互，如银行、事件、媒体、日历、旅行和天气。大多数领域包含多个不同的API，这些API具有重叠的功能但不同的接口，反映了常见的现实场景。该数据集的广泛注释可用于意图预测、槽填充、对话状态跟踪、策略模仿学习、语言生成和用户模拟学习等任务，用于开发大规模虚拟助手。此外，数据集在评估集中包含未见过的领域和服务，以量化零样本或少量样本设置中的性能。

SGD-X 数据集包含原始SGD数据集中每个模式的5种语言变体。这些语言变体由数百名付费众包工作者编写。在SGD-X目录中，v1代表最接近原始模式的变体，v5代表在语言距离上最远的变体。要评估模型在SGD-X模式上的性能，必须使用generate_sgdx_dialogues.py脚本转换对话。

数据集转换

• 如何从原始数据获取转换后的数据：

  • 下载 dstc8-schema-guided-dialogue-master.zip。
  • 根据 https://github.com/google-research-datasets/dstc8-schema-guided-dialogue/issues/57 修改 sgd_x/generate_sgdx_dialogues.py。
  • 在 dstc8-schema-guided-dialogue-master 目录下运行 python -m sgd_x.generate_sgdx_dialogues，需要安装tensorflow。
  • 在当前目录下运行 python preprocess.py。

• 转换的主要变化：

  • 将原始的 act 转换为小写的 intent。
  • 为每个领域添加 count 槽，非分类槽，通过文本匹配查找范围。
  • 根据 intent 对 dialogue acts 进行分类。
  • 使用 | 连接多个值。
  • 保留 active_intent、requested_slots、service_call。

注释

• 对话行为、状态、数据库结果、服务调用、活动意图、请求槽。

支持的任务和排行榜

• NLU, DST, Policy, NLG, E2E

语言

• 英语

数据分割

分割	对话数	话语数	平均话语数	平均词数	平均领域数	分类槽匹配(状态)	分类槽匹配(目标)	分类槽匹配(对话行为)	非分类槽范围(对话行为)
训练	16142	329964	20.44	9.75	1.84	100	-	100	100
验证	2482	48726	19.63	9.66	1.84	100	-	100	100
测试	4201	84594	20.14	10.4	2.02	100	-	100	100
全部	22825	463284	20.3	9.86	1.87	100	-	100	100

45个领域：[Banks_15, Buses_15, Buses_25, Calendar_15, Events_15, Events_25, Flights_15, Flights_25, Homes_15, Hotels_15, Hotels_25, Hotels_35, Media_15, Movies_15, Music_15, Music_25, RentalCars_15, RentalCars_25, Restaurants_15, RideSharing_15, RideSharing_25, Services_15, Services_25, Services_35, Travel_15, Weather_15, Alarm_15, Banks_25, Flights_35, Hotels_45, Media_25, Movies_25, Restaurants_25, Services_45, Buses_35, Events_35, Flights_45, Homes_25, Media_35, Messaging_15, Movies_35, Music_35, Payment_15, RentalCars_35, Trains_15]

• 分类槽匹配：分类槽的值在可能的值中的百分比。
• 非分类槽范围：非分类槽的值有范围注释的百分比。

引用

@inproceedings{lee2022sgd, title={SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems}, author={Lee, Harrison and Gupta, Raghav and Rastogi, Abhinav and Cao, Yuan and Zhang, Bin and Wu, Yonghui}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, volume={36}, number={10}, pages={10938--10946}, year={2022} }

许可信息

CC BY-SA 4.0