Multidimensional Gold-Standard Dataset for Explanation Needs in App Reviews

Overview

This dataset provides a multidimensional gold-standard dataset for identifying and categorizing explanation needs in mobile app reviews. It contains app metadata, cleaned review candidate datasets, manual annotation exports, consolidated gold-standard labels, interrater-agreement files, and reusable training, validation, and test splits derived from the gold-standard annotations where applicable.

The dataset is intended to support research on explanation need detection, requirements engineering, app review analysis, software engineering, and app-review classification using machine learning and large language models.

Dataset contents

The dataset is organized as follows:

data/01_app_metadata/: raw and cleaned app metadata collected from Apple App Store and Google Play Store.

data/02_cleaned_review_candidates/: cleaned and quality-controlled English review candidate datasets, including reviews with explanation-need indicators, mixed review candidates, and reviews without explanation-need indicators.

data/03_annotations/: manual annotation data, including individual rater annotations, interrater-agreement files, and consolidated gold-standard labels.

data/04_training_splits/: prepared training, validation, and test splits for explanation-need marking and taxonomy classification tasks, where applicable.

docs/: documentation files describing the taxonomy, label schema, annotation guidelines, preprocessing pipeline, and file organization.

The cleaned review candidate datasets in data/02_cleaned_review_candidates/ are intermediate candidate pools and should not be interpreted as final gold-standard labels. In particular, en_explanation_mixed contains candidate reviews used during the preparatory annotation and calibration phase, including reviews from the independent practice annotation round. These reviews were used to establish a shared understanding of the taxonomy and to identify early disagreements. They are not part of the final 5,004-review gold-standard dataset. The final gold-standard labels are provided under data/03_annotations/gold_labels/.

Annotation and taxonomy

The dataset distinguishes explicit explanation needs, implicit explanation needs, and reviews without explanation needs. For reviews containing explanation needs, the dataset further supports multidimensional categorization using the following taxonomy dimensions:

Time Aspect

Unexpected System Behavior - Bug

Software Feature

System Aspect

The human-readable taxonomy is provided in docs/taxonomy.md. A machine-readable label schema is provided in docs/label_schema.json. Annotation rules are documented in docs/annotation_guidelines.md.

File formats

The dataset uses open and widely supported formats:

.csv for tabular data,

.parquet for efficient tabular storage,

.json and .jsonl for structured metadata and training examples,

.xlsx for annotation exports and agreement sheets.

Suggested starting points

Use data/03_annotations/gold_labels/ to inspect the final gold-standard annotations.

Use data/03_annotations/rater_annotations/ to inspect individual rater annotations.

Use data/03_annotations/interrater_agreement/ to inspect interrater-agreement files.

Use data/04_training_splits/ to reuse the prepared training, validation, and test splits where applicable.

Use docs/taxonomy.md and docs/label_schema.json to understand the label structure.

Version information

Dataset timestamp: 2024-12-20T19-55-42.075Z

App metadata timestamp: 2024-11-30T23-33-36.178Z

Release date: 2026-05-23

Version: 1.0

Creator

Martin Obaidi

License

This dataset is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). You are free to share and adapt the material, including for commercial purposes, provided that appropriate credit is given.

Citation

If you use this dataset, please cite it as:

Martin Obaidi. Multidimensional Gold-Standard Dataset for Explanation Needs in App Reviews. Zenodo. https://doi.org/10.5281/zenodo.20359756

概览
本数据集为识别与归类移动应用评论中的解释需求提供了多维度的金标准数据集（gold-standard dataset）。数据集包含应用元数据、清洗后的评论候选集、人工标注导出文件、整合后的金标准标签（gold-standard labels）、标注者间一致性文件，以及从金标准标注中衍生的可复用训练、验证与测试划分（如适用）。

本数据集旨在支持解释需求检测、需求工程、应用评论分析、软件工程，以及基于机器学习与大语言模型（Large Language Model）的应用评论分类相关研究。

数据集内容
本数据集的组织形式如下：
- `data/01_app_metadata/`：从苹果应用商店（Apple App Store）与谷歌应用商店（Google Play Store）采集的原始与清洗后的应用元数据。
- `data/02_cleaned_review_candidates/`：经过清洗与质量管控的英文评论候选集，涵盖带解释需求标识的评论、混合评论候选集，以及无解释需求标识的评论。
- `data/03_annotations/`：人工标注数据，包含单标注者标注结果、标注者间一致性文件，以及整合后的金标准标签。
- `data/04_training_splits/`：针对解释需求标记与分类体系分类任务（如适用）准备的训练、验证与测试划分。
- `docs/`：用于描述分类体系、标签范式、标注指南、预处理流程与文件组织的文档文件。

`data/02_cleaned_review_candidates/`中的清洗后评论候选集为中间候选池，不应被视为最终金标准标签。特别地，`en_explanation_mixed`包含预备标注与校准阶段使用的候选评论，包括独立练习标注轮次的评论。此类评论用于建立对分类体系的共同理解，并识别早期分歧，不属于最终包含5004条评论的金标准数据集。最终金标准标签存放于`data/03_annotations/gold_labels/`。

标注与分类体系
本数据集区分显性解释需求、隐性解释需求与无解释需求的评论。对于包含解释需求的评论，本数据集进一步支持基于以下分类体系维度的多维度归类：
1. 时间维度（Time Aspect）
2. 非预期系统行为-缺陷（Unexpected System Behavior - Bug）
3. 软件功能（Software Feature）
4. 系统维度（System Aspect）

人类可读的分类体系详见`docs/taxonomy.md`，机器可读的标签范式详见`docs/label_schema.json`，标注规则详见`docs/annotation_guidelines.md`。

文件格式
本数据集采用开放且广泛支持的格式：
- 用于表格数据的`.csv`格式；
- 用于高效表格存储的`.parquet`格式；
- 用于结构化元数据与训练样本的`.json`与`.jsonl`格式；
- 用于标注导出与一致性表格的`.xlsx`格式。

建议使用起点
可通过以下路径开展相关研究：
1. 访问`data/03_annotations/gold_labels/`以查看最终金标准标注结果；
2. 访问`data/03_annotations/rater_annotations/`以查看单标注者的标注结果；
3. 访问`data/03_annotations/interrater_agreement/`以查看标注者间一致性文件；
4. 访问`data/04_training_splits/`以复用已准备好的训练、验证与测试划分（如适用）；
5. 访问`docs/taxonomy.md`与`docs/label_schema.json`以了解标签结构。

版本信息
- 数据集时间戳：2024-12-20T19-55-42.075Z
- 应用元数据时间戳：2024-11-30T23-33-36.178Z
- 发布日期：2026-05-23
- 版本：1.0

创建者
Martin Obaidi

许可协议
本数据集采用知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International License，CC BY 4.0）进行许可。您可自由共享与改编本材料，包括用于商业用途，但需给予适当署名。

引用说明
若您使用本数据集，请按以下格式引用：
Martin Obaidi. 面向应用评论解释需求的多维度金标准数据集（Multidimensional Gold-Standard Dataset for Explanation Needs in App Reviews）. Zenodo. https://doi.org/10.5281/zenodo.20359756