argument_quality_ranking_30k

# Dataset Card for Argument-Quality-Ranking-30k Dataset

## Table of Contents

- [Dataset Summary](#dataset-summary)
- [Argument Quality Ranking](#argument-quality-ranking)
- [Argument Topic](#argument-topic)
- [Dataset Collection](#dataset-collection)
- [Argument Collection](#argument-collection)
- [Quality and Stance Labeling](#quality-and-stance-labeling)
- [Dataset Structure](#dataset-structure)
- [Quality Labels](#quality-labels)
- [Stance Labels](#stance-labels)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Summary

### Argument Quality Ranking

The dataset contains 30,497 crowd-sourced arguments for 71 debatable topics labeled for quality and stance, split into train, validation and test sets.

The dataset was originally published as part of our paper: [A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis](https://arxiv.org/abs/1911.11408).

### Argument Topic

This subset contains 9,487 of the arguments only with their topics with a different train-validation-test split. Usage of this subset TBA.

## Dataset Collection

### Argument Collection

For the purpose of collecting arguments for this dataset we conducted a crowd annotation task. We selected 71 common controversial topics for which arguments were collected (e.g., We should abolish capital punishment).
Annotators were presented with a single topic each time, and asked to contribute one supporting and one contesting argument for it, requiring arguments to be written using original language. To motivate high-quality contributions, contributors were informed they will receive extra payment for high quality arguments, as determined by the subsequent argument quality labeling task.
It was explained that an argument will be considered as a high quality one, if a person preparing a speech on the topic will be likely to use this argument as is in her speech.
We place a limit on argument length - a minimum of 35 characters and a maximum of 210 characters. In total, we collected 30,497 arguments from 280 contributors, each contributing no more than 6 arguments per topic.

### Quality and Stance Labeling

Annotators were presented with a binary question per argument, asking if they would recommend a friend to use that argument as is in a speech supporting/contesting the topic, regardless of personal opinion.
In addition, annotators were asked to mark the stance of the argument towards the topic (pro or con).
10 annotators labeled each instance.

## Dataset Structure

Each instance contains a string argument, a string topic, and quality and stance scores:
* WA - the quality label according to the weighted-average scoring function
* MACE-P - the quality label according to the MACE-P scoring function
* stance_WA - the stance label according to the weighted-average scoring function
* stance_WA_conf - the confidence in the stance label according to the weighted-average scoring function

### Quality Labels

For an explanation of the quality labels presented in columns WA and MACE-P, please see section 4 in the paper.

### Stance Labels

There were three possible annotations for the stance task: 1 (pro), -1 (con) and 0 (neutral). The stance_WA_conf column refers to the weighted-average score of the winning label. The stance_WA column refers to the winning stance label itself.

## Licensing Information

The datasets are released under the following licensing and copyright terms:
* (c) Copyright [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Copyrights#Reusers.27_rights_and_obligations)
* (c) Copyright IBM 2014. Released under [CC-BY-SA 3.0](http://creativecommons.org/licenses/by-sa/3.0/)

## Citation Information

```
@article{DBLP:journals/corr/abs-1911-11408,
author = {Shai Gretz and
Roni Friedman and
Edo Cohen{-}Karlik and
Assaf Toledo and
Dan Lahav and
Ranit Aharonov and
Noam Slonim},
title = {A Large-scale Dataset for Argument Quality Ranking: Construction and
Analysis},
journal = {CoRR},
volume = {abs/1911.11408},
year = {2019},
url = {http://arxiv.org/abs/1911.11408},
eprinttype = {arXiv},
eprint = {1911.11408},
timestamp = {Tue, 03 Dec 2019 20:41:07 +0100},
biburl = {https://dblp.org/rec/journals/corr/abs-1911-11408.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
```

# Argument-Quality-Ranking-30k 数据集卡片（Dataset Card）

## 目录
- [数据集摘要](#dataset-summary)
- [论证质量排序（Argument Quality Ranking）](#argument-quality-ranking)
- [论证主题（Argument Topic）](#argument-topic)
- [数据集收集](#dataset-collection)
- [论证收集](#argument-collection)
- [质量与立场标注](#quality-and-stance-labeling)
- [数据集结构](#dataset-structure)
- [质量标签](#quality-labels)
- [立场标签](#stance-labels)
- [授权信息](#licensing-information)
- [引用信息](#citation-information)

## 数据集摘要

### 论证质量排序（Argument Quality Ranking）
该数据集包含针对71个争议性主题的30497条众包论证，已标注质量与立场信息，并划分为训练集、验证集与测试集。
本数据集最初作为我们的论文《面向论证质量排序的大规模数据集：构建与分析（A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis）》（https://arxiv.org/abs/1911.11408）的一部分发布。

### 论证主题（Argument Topic）
本子集包含9487条仅附带主题的论证，采用了不同的训练-验证-测试划分方式。该子集的使用方式待公布（TBA）。

## 数据集收集

### 论证收集（Argument Collection）
为收集本数据集的论证内容，我们开展了众包标注任务。我们筛选出71个常见争议主题，并针对这些主题收集论证（例如：我们应当废除死刑）。
每次向标注者展示单个主题，并要求其提交1条支持性论证与1条反驳性论证，且论证需使用其母语撰写。为激励高质量贡献，我们告知标注者，后续论证质量标注任务将判定优质论证，并为其提供额外报酬。
我们解释道，若某一针对该主题准备演讲的人员有可能直接在其演讲中使用该论证，则该论证将被视为高质量论证。
我们对论证长度设置了限制：最少35个字符，最多210个字符。最终，我们从280位贡献者处共收集到30497条论证，每位贡献者针对单个主题提交的论证不超过6条。

### 质量与立场标注（Quality and Stance Labeling）
针对每条论证，标注者需回答一个二元问题：无论个人观点如何，他们是否会推荐朋友在支持/反驳该主题的演讲中直接使用该论证。
此外，标注者需标记该论证针对该主题的立场（支持（pro）或反对（con））。
每条实例由10位标注者进行标注。

## 数据集结构

每个实例包含字符串形式的论证、字符串形式的主题，以及质量与立场得分：
* WA：基于加权平均（weighted-average）评分函数得到的质量标签
* MACE-P：基于MACE-P评分函数得到的质量标签
* stance_WA：基于加权平均评分函数得到的立场标签
* stance_WA_conf：基于加权平均评分函数得到的立场标签置信度

### 质量标签（Quality Labels）
关于列WA与MACE-P中质量标签的详细解释，请参见论文第4节。

### 立场标签（Stance Labels）
立场任务共有三种可能的标注结果：1（支持（pro））、-1（反对（con））与0（中立）。stance_WA_conf列指代获胜标签的加权平均得分，而stance_WA列指代获胜的立场标签本身。

## 授权信息（Licensing Information）
本数据集遵循以下授权与版权条款：
* (c) 版权归维基百科（Wikipedia）所有，详见https://en.wikipedia.org/wiki/Wikipedia:Copyrights#Reusers.27_rights_and_obligations
* (c) 版权归国际商业机器公司（IBM）2014年所有。根据CC-BY-SA 3.0（http://creativecommons.org/licenses/by-sa/3.0/）协议发布。

## 引用信息（Citation Information）

@article{DBLP:journals/corr/abs-1911-11408,
author = {Shai Gretz and
Roni Friedman and
Edo Cohen{-}Karlik and
Assaf Toledo and
Dan Lahav and
Ranit Aharonov and
Noam Slonim},
title = {A Large-scale Dataset for Argument Quality Ranking: Construction and
Analysis},
journal = {CoRR},
volume = {abs/1911.11408},
year = {2019},
url = {http://arxiv.org/abs/1911.11408},
eprinttype = {arXiv},
eprint = {1911.11408},
timestamp = {Tue, 03 Dec 2019 20:41:07 +0100},
biburl = {https://dblp.org/rec/journals/corr/abs-1911.11408.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}