Dataset: Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board

This is the dataset released with the paper titled: "Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board". The dataset is a single Newline delimited JSON file. Each line in the file consists of a JSON object which is a full 4chan /pol/ thread. The JSON objects contain all the key/values returned by the 4chan API, along with three additional keys (entities, perspectives, and extracted_poster_id). For each JSON object we complement the data with the list of the named entities we detect for each post, using the spaCy Python library. In addition, for each post we add scores returned by the Google’s Perspective API, and more specifically seven scores in the [0; 1] interval. For the detailed description of every key in the JSON structure, along with the type of the value, please read the readme.pdf file provided with this dataset. If you find our dataset useful, please cite our paper: @article{papasavva2020raiders, title={Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board}, author={Antonis Papasavva, Savvas Zannettou, Emiliano De Cristofaro, Gianluca Stringhini, Jeremy Blackburn}, journal={14th International AAAI Conference On Web And Social Media (ICWSM), 2020}, year={2020} } How to extract the data: Note that the data is compressed. See the instructions below on how to extract the data: Linux and Mac Step 1: Open a terminal window and navigate to the path where the file pol_0616-1119_labeled.tar.zst is located. Step2: Run the following command: unzstd pol_0616-1119_labeled.tar.zst The above command will result in a file named pol_0616-1119_labeled.tar. (in the same directory) Step 3: Again, from your terminal window, run this command: tar -xvf pol_0616-1119_labeled.tar When the above command finishes, you will get (in the same directory) the extracted data - a file named pol_062016-112019_labeled.ndjson. Windows There are many applications that can be used to extract this data on Windows available online. The authors cannot recommend specific applications. Note that the file is compressed twice so you will need to perform the data extraction twice - once on the downloaded file, and once on the file that was extracted from the downloaded file. Please do not hesitate to contact the author of this study in case you face any problem at: [e-mail]本数据集配套发表于题为《Raiders of the Lost Kek：3.5年扩增的4chan /pol/板块发帖数据》（Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board）的学术论文。本数据集为单个换行分隔JSON（Newline Delimited JSON，NDJSON）文件，文件中每一行均为一个完整的4chan /pol/板块帖子对应的JSON对象。该JSON对象包含4chan API返回的全部键值对，同时新增三个额外键：entities、perspectives与extracted_poster_id。 针对每个JSON对象，我们借助Python库spaCy（spaCy）对单条帖子开展命名实体识别，并将识别得到的命名实体列表补充至数据中。此外，我们还补充了Google Perspective API（Perspective API）返回的评分指标，共7项，取值范围均处于[0, 1]区间内。 若需了解JSON结构中各键的详细说明及其对应值的类型，请查阅本数据集附带的readme.pdf文件。 若您认为本数据集对您的研究有所帮助，请引用以下论文： @article{papasavva2020raiders, title={Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Board}, author={Antonis Papasavva, Savvas Zannettou, Emiliano De Cristofaro, Gianluca Stringhini, Jeremy Blackburn}, journal={14th International AAAI Conference On Web And Social Media (ICWSM), 2020}, year={2020} } ### 数据提取方法 注：本数据集已压缩，请按照以下步骤完成解压： #### Linux与Mac系统 步骤1：打开终端窗口并导航至压缩文件`pol_0616-1119_labeled.tar.zst`所在的目录。 步骤2：执行以下命令：`unzstd pol_0616-1119_labeled.tar.zst` 执行上述命令后，将在当前目录下生成名为`pol_0616-1119_labeled.tar`的解压文件。 步骤3：再次在终端中执行以下命令：`tar -xvf pol_0616-1119_labeled.tar` 命令执行完成后，当前目录下将得到解压后的数据文件`pol_062016-112019_labeled.ndjson`。 #### Windows系统 网络上有多款工具可用于解压本数据集，本文作者不推荐特定工具。请注意本数据集经过两次压缩，因此需执行两次解压操作：先解压下载得到的压缩文件，再解压首次解压得到的文件。 若您在使用过程中遇到任何问题，请联系本研究的作者邮箱：[e-mail]