Replication Data for: Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression

This dataset provides code, data, and instructions for replicating the analysis of Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression published in OpenSym 2021 (link to come). The paper introduces a method for transforming scores from the ORES quality models into a single dimensional measure of quality amenable for statistical analysis that is well-calibrated to a dataset. The purpose is to improve the validity of research into article quality through more precise measurement. The code and data for replicating the paper are found in this dataverse repository. If you wish to use method on a new dataset, you should obtain the actively maintaned version of the code from this git repository. If you attempt to replicate part of this repository please let me know via an email to [e-mail]. Replicating the Analysis from the OpenSym Paper This project analyzes a sample of articles with quality labels from the English Wikipedia XML dumps from March 2020. Copies of the dumps are not provided in this dataset. They can be obtained via https://dumps.wikimedia.org/. Everything else you need to replicate the project (other than a sufficiently powerful computer) should be available here. The project is organized into stages. The prerequisite data files are provided at each stage so you do not need to rerun the entire pipeline from the beginning, which is not easily done without a high-performance computer. If you start replicating at an intermediate stage, this should overwrite the inputs to the downstream stages. This should make it easier to verify a partial replication. To help manage the size of the dataverse, all code files are included in code.tar.gz. Extracting this with tar xzvf code.tar.gz is the first step. Getting Set Up You need a version of R >= 4.0 and a version of Python >= 3.7.8. You also need a bash shell, tar, gzip, and make installed as they should be installed on any Unix system. To install brms you need a working C++ compiler. If you run into trouble see the instructions for installing Rstan. The datasets were built on CentOS 7, except for the ORES scoring which was done on Ubuntu 18.04.5 and building which was done on Debian 9. The RemembR and pyRembr projects provide simple tools for saving intermediate variables for building papers with LaTex. First, extract the articlequality.tar.gz, RemembR.tar.gz and pyRembr.tar.gz archives. Then, install the following: Python Packages Running the following steps in a new Python virtual environment is strongly recommended. Run pip3 install -r requirements.txt to install the Python dependencies. Then navigate into the pyRembr directory and run python3 setup.py install. R Packages Run Rscript install_requirements.R to install the necessary R libraries. If you run into trouble installing brms see the instructions on Drawing a Sample of Labeled Articles I provide steps and intermediate data files for replicating the sampling of labeled articles. The steps in this section are quite computationally intensive. Those only interested in replicating the models and analyses should skip this section. Extracting Metadata from Wikipedia Dumps Metadata from the Wikipedia dumps is required for calibrating models to the revision and article levels of analysis. You can use the wikiq Python script from the mediawiki dump tools git repository to extract metadata from the XML dumps as TSV files. The version of wikiq that was used is provided here. Running Wikiq on a full dump of English Wikipedia in a reasonable amount of requires considerable computing resources. For this project, Wikiq was run on Hyak a high performance computer at the University of Washington. The code for doing so is highly speicific to Hyak. For transparency and in case it helps others using similar academic computers this code is included in WikiqRunning.tar.gz. A copy of the wikiq output is included in this dataset in the multi-part archive enwiki202003-wikiq.tar.gz. To extract this archive, download all the parts and then run cat enwiki202003-wikiq.tar.gz* > enwiki202003-wikiq.tar.gz && tar xvzf enwiki202003-wikiq.tar.gz. Obtaining Quality Labels for Articles We obtain up-to-date labels for each article using the articlequality python package included in articlequality.tar.gz. The XML dumps are also the input to this step, and while it does not require a great deal of memory, a powerful computer (we used 28 cores) is helpful so that it completes in a reasonable amount of time. extract_quality_labels.sh runs the command to extract the labels from the xml dumps. The resulting files have the format data/enwiki-20200301-pages-meta-history*.xml-p*.7z_article_labelings.json and are included in this dataset in the archive enwiki202003-article_labelings-json.tar.gz. Taking a Sample of Quality Labels I used Apache Spark to merge the metadata from Wikiq with the quality labels and to draw a sample of articles where each quality class is equally represented. To improve the efficiency of this merge, I first converted the wikiq tsv outputs into the parquet columnar format using the script wikiq_to_parquet.py. The parquet files are in this dataset in the multi-part archive enwiki202003-wikiq-parquet.tar.gz. To extract this archive, download all the parts and then run cat enwiki202003-wikiq-parquet.tar.gz* > enwiki202003-wikiq-parquet.tar.gz && tar xvzf enwiki202003-wikiq-parquet.tar.gz. It also saves the counts of revisions and articles by quality class to remember_sample_quality_labels.RDS for computing the weights. Run python3 sample_training_labels.py to draw the sample which is written to 20200301_article_labelings_sample.json and additional metadata is saved to training_quality_labels.RDS. Obtaining ORES Scores Scoring the sample of labeled articles requires an internet connection to Wikipedia. The Python script ores_scores_sample.py uses the revscoring package to obtain quality scores for the sample. The bash script ores_score_sample.sh runs the script. The scores are output to scored_article_sample.feather, the same data is provided in scored_article_sample.csv. This is the dataset used in the remaining analysis. If you run this step you should edit the user_agent variable in ores_score_sample.py. Computing Weighted Datasets load_data.R reads remember_sample_quality_labels.RDS and scored_article_sample.feather to finialize the weighted dataset. Each of the R scripts source this file to load the dataset. Fitting Ordinal Regression Models All three bayesian regression models are fit in ordinal_quality_models.R. The models are saved in the models.tar.gz archive. The scripts fit the models in parallel and will use 12 cores. Interpreting the Models There are 3 separate scripts for analyzing each of the three models and the ORES model: analyze_quality_models_articles.R, analyze_quality_models_unweighted.R and analyze_quality_models_revisions.R. For convenience, the script run_ordinal_quality.sh runs the R scripts for fitting and interpreting all the models. These scripts output 4 .RDS files that contain data for use by knitr to insert the numbers, graphs, and tables in to the paper. Building the Paper Extract paper.tar.gz and run make in the resulting directory to build the paper using the data stored in the provided .RDS files. Calibrating a Model for a New Sample TK本数据集提供代码、数据与操作指南，用于复现发表于"OpenSym 2021"的论文《通过序数回归扩展ORES以单维度衡量维基百科条目质量》（*Measuring Wikipedia Article Quality in One Dimension by Extending ORES with Ordinal Regression*）的分析工作，论文链接待公布。该论文提出了一种方法，可将ORES（在线修订评估服务，Online Revision Evaluation Service）质量模型生成的评分转换为单维度质量衡量指标，适配统计分析且可针对特定数据集实现良好校准，旨在通过更精准的测量提升条目质量相关研究的效度。 本数据集的Dataverse数据共享仓库中提供了复现该论文所需的代码与数据。若希望将该方法应用于新数据集，请从该Git仓库获取代码的主动维护版本。若尝试复现本仓库的部分内容，请通过邮件[e-mail]告知笔者。 ### 复现OpenSym 2021论文的分析流程 本项目针对2020年3月的英文维基百科XML转储文件中的带质量标签的条目样本开展分析。本数据集未提供转储文件副本，可通过https://dumps.wikimedia.org/ 获取。除性能足够的计算机外，复现本项目所需的其余资源均可在此数据集中获取。 本项目按阶段组织，每个阶段均提供了前置数据文件，因此无需从头运行完整流程——若无高性能计算机，从头运行完整流程难度较大。若从中间阶段开始复现，该阶段的输入会覆盖下游阶段的输入，这有助于简化部分复现的验证工作。为控制Dataverse的存储空间，所有代码文件均打包在`code.tar.gz`中，第一步需使用`tar xzvf code.tar.gz`进行解压。 #### 环境配置 需使用R >= 4.0版本与Python >= 3.7.8版本，同时需要Unix系统默认预装的bash shell、tar、gzip与make工具。安装brms（基于Stan的贝叶斯回归模型包）需具备可用的C++编译器，若遇到安装问题，请参考Rstan的安装指南。 本数据集的构建环境为CentOS 7，其中ORES评分环节基于Ubuntu 18.04.5，编译环节基于Debian 9。 RemembR与pyRembr项目提供了简易工具，可用于保存中间变量以辅助基于LaTeX的论文撰写工作。首先需解压`articlequality.tar.gz`、`RemembR.tar.gz`与`pyRembr.tar.gz`归档文件。 ##### Python依赖安装 强烈建议在全新的Python虚拟环境中执行以下步骤。运行`pip3 install -r requirements.txt`安装Python依赖包，随后进入`pyRembr`目录并执行`python3 setup.py install`完成安装。 ##### R依赖安装 运行`Rscript install_requirements.R`安装所需的R库。若安装brms时遇到问题，请参考官方安装指南。 #### 带标签条目样本的抽取 本节提供了复现带标签条目抽样的步骤与中间数据文件，该环节计算量较大，仅需复现模型与分析的用户可跳过本节。 ##### 从维基百科转储文件提取元数据 需从维基百科转储文件中提取元数据，以完成针对修订版本与条目的模型校准。可使用mediawiki转储工具Git仓库中的`wikiq` Python脚本，将XML转储文件的元数据提取为TSV（制表符分隔值，Tab-Separated Values）格式文件。本数据集提供了所用版本的`wikiq`工具。 若要在合理时间内完成完整英文维基百科转储的处理，需要大量计算资源。本项目中，`Wikiq`运行于华盛顿大学的高性能计算机Hyak，所用代码高度适配Hyak环境。为保证透明度并帮助其他使用同类学术计算平台的用户，相关代码已包含在`WikiqRunning.tar.gz`中。 本数据集的多部分归档`enwiki202003-wikiq.tar.gz`中包含了`wikiq`的输出结果。如需解压该归档，请下载所有分卷文件后执行：`cat enwiki202003-wikiq.tar.gz* > enwiki202003-wikiq.tar.gz && tar xvzf enwiki202003-wikiq.tar.gz`。 ##### 获取条目质量标签 我们使用`articlequality.tar.gz`中包含的`articlequality` Python包为每个条目获取最新的质量标签。本步骤同样以XML转储文件作为输入，虽然该步骤对内存要求不高，但使用高性能计算机（本次使用了28核服务器）可显著缩短运行时间。 运行`extract_quality_labels.sh`即可执行从XML转储文件提取标签的命令。生成的文件格式为`data/enwiki-20200301-pages-meta-history*.xml-p*.7z_article_labelings.json`，本数据集的归档`enwiki202003-article_labelings-json.tar.gz`中包含了这些文件。 ##### 抽取质量标签样本 我使用Apache Spark（分布式计算框架）合并了`Wikiq`的元数据与质量标签，并抽取了每个质量类别占比均等的条目样本。为提升合并效率，我首先使用脚本`wikiq_to_parquet.py`将`wikiq`的TSV输出转换为Parquet（列式存储格式）格式。Parquet文件存储在本数据集的多部分归档`enwiki202003-wikiq-parquet.tar.gz`中。如需解压该归档，请下载所有分卷文件后执行：`cat enwiki202003-wikiq-parquet.tar.gz* > enwiki202003-wikiq-parquet.tar.gz && tar xvzf enwiki202003-wikiq-parquet.tar.gz`。 该步骤还会将按质量类别统计的修订版本与条目数量保存至`remember_sample_quality_labels.RDS`，用于计算权重。运行`python3 sample_training_labels.py`即可抽取样本，结果将写入`20200301_article_labelings_sample.json`，额外的元数据将保存至`training_quality_labels.RDS`。 ##### 获取ORES评分 为带标签的样本条目生成评分需要连接维基百科的互联网接口。Python脚本`ores_scores_sample.py`使用revscoring包为样本获取质量评分，可通过bash脚本`ores_score_sample.sh`运行该脚本。评分结果将输出至`scored_article_sample.feather`，同时也以`scored_article_sample.csv`格式提供。该数据集即为后续分析所用的核心数据集。若运行该步骤，请编辑`ores_score_sample.py`中的`user_agent`变量。 ##### 计算加权数据集 `load_data.R`会读取`remember_sample_quality_labels.RDS`与`scored_article_sample.feather`以生成最终的加权数据集。所有R脚本均会调用该文件加载数据集。 ##### 拟合序数回归模型 三个贝叶斯回归模型均在`ordinal_quality_models.R`中完成拟合，模型将保存至`models.tar.gz`归档中。该脚本会并行运行，默认使用12个核心。 ##### 模型解读 针对三个模型与ORES模型，分别提供了三个独立的分析脚本：`analyze_quality_models_articles.R`、`analyze_quality_models_unweighted.R`与`analyze_quality_models_revisions.R`。为方便使用，脚本`run_ordinal_quality.sh`可一键运行所有模型拟合与解读的R脚本。 这些脚本将输出4个.RDS文件，包含供knitr（R语言动态文档生成工具）将数值、图表与表格插入论文所需的数据。 ##### 构建论文 解压`paper.tar.gz`并进入生成的目录后执行`make`命令，即可使用提供的.RDS文件中的数据完成论文的编译。 ##### 为新样本校准模型 （原文标注TK，内容待补充）