Replication Package - How Do Requirements Evolve During Elicitation? An Empirical Study Combining Interviews and App Store Analysis

This is the replication package for the paper titled "How Do Requirements Evolve During

Elicitation? An Empirical Study Combining Interviews and App Store Analysis", by Alessio Ferrari, Paola Spoletini and Sourav Debnath.

The package contains the following folders and files. 

**/Experiment Material**

This folder contains the material used for the experiment, and provided to the participants.

In particular, it includes the following files:

- Happy CampingTM_briefdescription.pdf/docx: brief description of the product for which requirements need to be elicited

- Hw_description.pdf/docx: desciption of the tasks to be performed by the participants

- Modeling_Intro_Slides.pdf: introductory slides to modelling for requirements engineering

- Self-assessment Questionnaire.pdf: first questionnaire to self-assess the mistakes, from the SaPeer method (https://doi.org/10.1007/s00766-020-00334-0) 

- Self-assessment Questionnaire (Second Interview).pdf: second questionnare to self-assess the mistakes, from the Sapeer method

**/R-analysis**

This is a folder containing all the R implementations of the the statistical tests included in the paper, together with the source .csv file used to produce the results. Each R file has the same title as the associated .csv file. The titles of the files reflect the RQs as they appear in the paper. The association between R files and Tables in the paper is as follows:

- RQ1-1-analyse-story-rates.R: Tabe 1, user story rates 

- RQ1-1-analyse-role-rates.R: Table 1, role rates

- RQ1-2-analyse-story-category-phase-1.R: Table 3, user story category rates in phase 1 compared to original rates

- RQ1-2-analyse-role-category-phase-1.R: Table 5, role category rates in phase 1 compared to original rates

- RQ2.1-analysis-app-store-rates-phase-2.R: Table 8, user story and role rates in phase 2

- RQ2.2-analysis-percent-three-CAT-groups-ph1-ph2.R: Table 9, comparison of the categories of user stories in phase 1 and 2

- RQ2.2-analysis-percent-two-CAT-roles-ph1-ph2.R: Table 10, comparison of the categories of roles in phase 1 and 2.  

The .csv files used for statistical tests are also used to produce boxplots. The association betwee boxplot figures and files is as follows. 

- RQ1-1-story-rates.csv: Figure 4 

- RQ1-1-role-rates.csv: Figure 5

- RQ1-2-categories-phase-1.csv: Figure 8

- RQ1-2-role-category-phase-1.csv: Figure 9

- RQ2-1-user-story-and-roles-phase-2.csv: Figure 13

- RQ2.2-percent-three-CAT-groups-ph1-ph2.csv: Figure 14

- RQ2.2-percent-two-CAT-roles-ph1-ph2.csv: Figure 17

- IMG-only-RQ2.2-us-category-comparison-ph1-ph2.csv: Figure 15

- IMG-only-RQ2.2-frequent-roles.csv: Figure 18

NOTE: The last two .csv files do not have an associated statistical tests, but are used solely to produce boxplots.

**/Data-Analysis**

This folder contains all the data used to answer the research questions. 

**RQ1.xlsx**: includes all the data associated to RQ1 subquestions, two tabs for each subquestion (one for user stories and one for roles). The names of the tabs are self-explanatory of their content.

**RQ2.1.xlsx**: includes all the data for the RQ1.1 subquestion. Specifically, it includes the following tabs:

* Data Source-US-category: for each category of user story, and for each analyst, there are two lines. 

The first one reports the number of user stories in that category for phase 1, and the second one reports the

number of user stories in that category for phase 2, considering the specific analyst. 

* Data Source-role: for each category of role, and for each analyst, there are two lines. 

The first one reports the number of user stories in that role for phase 1, and the second one reports the

number of user stories in that role for phase 2, considering the specific analyst. 

* RQ2.1 rates: reports the final rates for RQ2.1. 

NOTE: The other tabs are used to support the computation of the final rates.

**RQ2.2.xlsx**: includes all the data for the RQ2.2 subquestion. Specifically, it includes the following tabs:

* Data Source-US-category: same as RQ2.1.xlsx

* Data Source-role: same as RQ2.1.xlsx

* RQ2.2-category-group: comparison between groups of categories in the different phases, used to produce Figure 14

* RQ2.2-role-group: comparison between role groups in the different phases, used to produce Figure 17

* RQ2.2-specific-roles-diff: difference between specific roles, used to produce Figure 18

**NOTE:** the other tabs are used to support the computation of the values reported in the tabs above. 

**RQ2.2-single-US-category.xlsx**: includes the data for the RQ2.2 subquestion associated to single categories of user stories.

A separate tab is used given the complexity of the computations. 

* Data Source-US-category: same as RQ2.1.xlsx

* Totals: total number of user stories for each analyst in phase 1 and phase 2

* Results-Rate-Comparison: difference between rates of user stories in phase 1 and phase 2, used to produce the file

"img/IMG-only-RQ2.2-us-category-comparison-ph1-ph2.csv", which is in turn used to produce Figure 15

* Results-Analysts: number of analysts using each novel category produced in phase 2, used to produce Figure 16.

NOTE: the other tabs are used to support the computation of the values reported in the tabs above. 

**RQ2.3.xlsx**: includes the data for the RQ2.3 subquestion. Specifically, it includes the following tabs:

* Data Source-US-category: same as RQ2.1.xlsx

* Data Source-role: same as RQ2.1.xlsx

* RQ2.3-categories: novel categories produced in phase 2, used to produce Figure 19

* RQ2-3-most-frequent-categories: most frequent novel categories

**/Raw-Data-Phase-I**

The folder contains one Excel file for each analyst, s1.xlsx...s30.xlsx, plus the file of the original user stories with annotations (original-us.xlsx). Each file contains two tabs:

- Evaluation: includes the annotation of the user stories as existing user story in the original categories (annotated with "E"), novel user story in a certain category (refinement, annotated with "N"), and novel user story in novel category (Name of the category in column "New Feature"). **NOTE 1:** It should be noticed that in the paper the case "refinement" is said to be annotated with "R" (instead of "N", as in the files) to make the paper clearer and easy to read. 

- Roles: roles used in the user stories, and count of the user stories belonging to a certain role.

**/Raw-Data-Phaes-II**

The folder contains one Excel file for each analyst, s1.xlsx...s30.xlsx. Each file contains two tabs:

- Analysis: includes the annotation of the user stories as belonging to existing original 

category (X), or to categories introduced after interviews, or to categories introduced 

after app store inspired elicitation (name of category in "Cat. Created in PH1"), or to 

entirely novel categories (name of category in "New Category").

- Roles: roles used in the user stories, and count of the user stories belonging to a certain role.

**/Figures**

This folder includes the figures reported in the paper. The boxplots are generated from the 

data using the tool http://shiny.chemgrid.org/boxplotr/. The histograms and other plots are 

produced with Excel, and are also reported in the excel files listed above.