What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews - Replication Package

Emotion analysis from app reviews - Replication package<i>Please read the README file for a full description of the artifact.</i>📚 Summary of artifactThis artifact supports the replication of the study presented in the paper "What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews", accepted at the 33rd IEEE International Requirements Engineering 2025 conference. It provides a comprehensive framework for conducting fine-grained emotion analysis from mobile app reviews using both human and large language model (LLM)-based annotations.The artifact includes:<b>Input</b>: A dataset of user reviews, emotion annotation guidelines, and ground truth annotations from human annotators.<b>Process</b>: Scripts for generating emotion annotations via LLMs (GPT-4o, Mistral Large 2, and Gemini 2.0 Flash), splitting annotations into iterations, computing agreement metrics (e.g., Cohen’s Kappa), and evaluating correctness and cost-efficiency.<b>Output</b>: Annotated datasets (human and LLM-generated), agreement analyses, emotion statistics, and evaluation metrics including accuracy, precision, recall, and F1 score.The artifact was developed to ensure transparency, reproducibility, and extensibility of the experimental pipeline. It enables researchers to replicate, validate, or extend the emotion annotation process across different LLMs and configurations, contributing to the broader goal of integrating emotional insights into requirements engineering practices.Complementarily, the artifact is described using the NLP4RE ID-Card template . More details available at:Sallam Abualhaija, F. Basak Aydemir, Fabiano Dalpiaz, Davide Dell'Anna, Alessio Ferrari, Xavier Franch, and Davide Fucci. 2024. Replication in Requirements Engineering: The NLP for RE Case. ACM Trans. Softw. Eng. Methodol. 33, 6, Article 151 (July 2024), 33 pages. https://doi.org/10.1145/3658669🔎 Artifact LocationThe artifact is available at https://doi.org/10.6084/m9.figshare.28548638.📂 Description of ArtifactThe root folder of the artifact contains the following materials:<b>Quick Start Guide</b>: PDF containing a simplified description of the artifact for first-time users.<b>README</b>: README file containing the description of the artifact.<b>LICENSE</b>: License of the artifact and its resources (including code and datasets).<b>Annotation Guidelines</b>: PDF document containing the detailed, extended guidelines for emotion annotation.<b>Ground Truth</b>: XLSX and CSV files containing the dataset of app reviews annotated with emotions.<b>NLP4REIDCard</b>: PDF document containing the metadata of the NLP4RE ID-Card used to document the artifact within the scope of NLP4RE tools and artifacts.<b>Replication package</b>: ZIP file containing the code, datasets and evaluation reports generated through the research. This includes the following: -- <b>Literature review</b>: results from the literature review on opinion mining and emotion analysis within the context of software-based reviews. -- <b>Data</b>: data used in the study, including user reviews (input), human annotations (ground truth), and LLM-based annotations (generated by the assistants). -- <b>Code</b>: code used in the study, including the generative annotation, data processing, and evaluation.For more details, please refer to the README file.