Replication package for the paper: "A Study on the Pythonic Functional Constructs' Understandability"

Replication Package for "A Study on the Pythonic Functional Constructs' Understandability" to appear at ICSE 2024 Authors: Cyrine Zid, Fiorella Zampetti, Giuliano Antoniol, Massimiliano Di penta Article Preprint: https://mdipenta.github.io/files/ICSE24_funcExperiment.pdf License: GPL V3.0 This package contains several folders and files with code and data used in the study. Experiment-Material/ Contains the material used for the experiment, and, specifically, the following subdirectories: Google-Forms/ Contains (as PDF documents) the questionnaires submitted to the ten experimental groups. Task-Sources/ Contains, for each experimental group (G-1...G-10), the sources used to produce the Google Forms, and, specifically: - The cover letter (Letter.docx). - A directory for each experimental task (Lambda 1, Lambda 2, Comp 1, Comp 2, MRF 1, MRF 2, Lambda Comparison, Comp Comparison, MRF Comparison). Each directory contains: (i) the exercise text, the source code snippet, and its .png image to be used in the form. Code-Examples-Table1/ Contains the source code snippets used as objects of the study (the same you can find under "Task-Sources/"), named as reported in Table 1. Results' directory and files raw-results-RQ1/ Contains the raw results for RQ1. Specifically, the directory contains a subdirectory for each group (G1-G10). Each subdirectory contains: - For each user (named using their Prolific IDs, a directory containing, for each question (Q1-Q6) the produced python code (Qn.py) its output (QnR.txt) and its StdErr output (QnErr.txt). - "expected-outputs/": A directory containing the expected outputs for each task (Qn.txt). working-results/RQ1-RQ2-files-for-statistical-analysis/ Contains three .csv files used as input for conducting the statistical analysis and drawing the graphs for addressing the first two research questions of the study. Specifically: ConstructUsage.csv contains the declared frequency usage of the three functional constructs object of the study. This file is used to draw Figure 4. The file contains an entry for each participant, reporting the (text-coded) frequency of construct usage for Comprehension, Lambda, and MRF. RQ1.csv contains the collected data used for the mixed-effect logistic regression relating the use of functional constructs with the correctness of the change task, as well as the logistic regression relating the use of map/reduce/filter functions with the correctness of the change task. The csv file contains an entry for each answer provided by each subject, and features the following columns: Group: experimental group to which the participant is assigned User: user ID Time: task time in seconds Approvals: number of approvals on previous tasks performed on Prolific Student: whether the participant declared themselves as a student Section: section of the questionnaire (lambda, comp, or mrf) Construct: specific construct being presented (same as "Section" for lambda and comp, for mrf it says whether it is a map, reduce, or filter) Question: question id, from Q1 to Q6, indicate the ordering of the question MainFactor: main factor treatment for the given question - "f" for functional, "p" for procedural counterpart Outcome: TRUE if the task was correctly performed, FALSE otherwise Complexity: cyclomatic complexity of the construct (empty for mrf) UsageFrequency: usage frequency of the given construct RQ1Paired-RQ2.csv contains the collected data used for the ordinal logistic regression of the relationship between the perceived ease of understanding of the functional constructs and (i) participants' usage frequency, and (ii) constructs' complexity (except for map/reduce/filter). The file features a row for each participant, and the columns are the following: Group: experimental group to which the participant is assigned User: user ID Time: task time in seconds Approvals: number of approvals on previous tasks performed on Prolific Student: whether the participant declared themselves as a student LambdaF: result for the change task related to a lambda construct LambdaP: result for the change task related to the procedural counterpart of a lambda construct CompF: result for the change task related to a comprehension construct CompP: result for the change task related to the procedural counterpart of a comprehension construct MrfF: result for the change task related to an MRF construct MrfP: result for the change task related to the procedural counterpart of a MRF construct LambdaComp: perceived understandability level for the comparison task (RQ2) between a lambda and its procedural counterpart CompComp: perceived understandability level for the comparison task (RQ2) between a comprehension and its procedural counterpart MrfComp: perceived understandability level for the comparison task (RQ2) between a MRF and its procedural counterpart LambdaCompCplx: cyclomatic complexity of the lambda construct involved in the comparison task (RQ2) CompCompCplx: cyclomatic complexity of the comprehension construct involved in the comparison task (RQ2) MrfCompType: type of MRF construct (map, reduce, or filter) used in the comparison task (RQ2) LambdaUsageFrequency: self-declared usage frequency on lambda constructs CompUsageFrequency: self-declared usage frequency on comprehension constructs MrfUsageFrequency: self-declared usage frequency on MRF constructs LambdaComparisonAssessment: outcome of the manual assessment of the answer to the "check question" required for the lambda comparison ("yes" means valid, "no" means wrong, "moderate-chatgpt" and "extreme-chatgpt" are the results of GPTZero) CompComparisonAssessment: as above, but for comprehension MrfComparisonAssessment: as above, but for MRF working-results/inter-rater-RQ3-files/ This directory contains four .csv files used as input for computing the inter-rater agreement for the manual labeling used for addressing RQ3. Specifically, you will find one file for each functional construct, i.e., comprehension.csv, lambda.csv, and mrf.csv, and a different file used for highlighting the reasons why participants prefer to use the procedural paradigm, i.e., procedural.csv. working-results/RQ2ManualValidation.csv This file contains the results of the manual validation being done to sanitize the answers provided by our participants used for addressing RQ2. Specifically, we coded the behaviour description using four different levels: (i) correct ("yes"), (ii) somewhat correct ("partial"), (iii) wrong ("no"), and (iv) automatically generated. The file features a row for each participant, and the columns are the following: ID: ID we used to refer the participant in the paper's qualitative analysis Group: experimental group to which the participant is assigned ProlificID: user ID Comparison for lambda construct description: answer provided by the user for the lambda comparison task Final Classification: our assessment of the lambda comparison answer Comparison for comprehension description: answer provided by the user for the comprehension comparison task Final Classification: our assessment of the comprehension comparison answer Comparison for MRF description: answer provided by the user for the MRF comparison task Final Classification: our assessment of the MRF comparison answer working-results/RQ3ManualValidation.xlsx This file contains the results of the open coding applied to address our third research question. Specifically, you will find four sheets, one for each functional construct and one for the procedural paradigm. Each sheet reports the provided answers together with the categories assigned to them. Each sheet contains the following columns: - ID: ID we used to refer the participant in the paper's qualitative analysis - Group: experimental group to which the participant is assigned - ProlificID: user ID (as in the tables from the quantitative analysis)- : question asked to the user - Final Classification: The outcome of our categorization according to the taxonomy shown in Table 10. Scripts to reproduce the results, and instructions for running them FuncConstructs-Statistics.r This file contains an R script that you can reuse to re-run all the analyses conducted and discussed in the paper. FuncConstructs-Statistics.ipynb This file contains the code to re-execute all the analysis conducted in the paper as a Jupyter Notebook (using the R Kernel). run-analysis.sh This script can be used to run the R script FuncConstructs-Statistics.r using a Docker container (see Option 1 below). run-jupyter-container.sh This script can be used to run a local Jupyter server (with R kernel and all required packages) from a Docker container (see Option 3 below). How to Run the scripts There are four options to run the scripts. In all cases, one has first to open a shell terminal window (e.g., bash or sh in Unixes) in the replication package directory. For Windows, we suggest to use a Power Shell. Running the R script using Dockerized R installation: this is the simplest option, and it simply requires a running Docker engine. To produce the results, one has to run the shell script "run-analyses.sh" (e.g., by typing sh run-analysis.sh or simply ./run-analysis.shafter making it executable). This script will: Pull a docker image named mdipenta/rexp which contains an R installation with all required packages. Run R from the container created from the image and produce the paper's results under a directory named results/. Note: An alternative would be to run everything from inside the container, after running it in interactive mode. To this aim, please execute the following commands: docker run -v${PWD}:/data --rm -ti --name shell mdipenta/rexp:latest bash cd data R --no-save < FuncConstructs-Statistics.r After exiting the container, the "results" directory will be again populated with the study results. Running the R script from own R installation: this option works if one has an R installation already (or wants to use an R installation) without relying on the Docker image. The steps to be followed are: Uncomment the install.packages(..) instruction in the first lines of the script. This will allow for the installation of the required packages. Just run, from the current directory, the script FuncConstructs-Statistics.r, e.g., using the command cat FuncConstructs-Statistics.r | R --no-save Using the Jupyter Notebook using a Dockerized Jupyter lab with R kernel: this option allows for opening the Jupyter Notebook with all results without having to install Jupyter with the R kernel, nor all the required R packages. The steps required are: Run the ./run-jupyter-container.sh. It will download the mdipenta/myjupyter image and run Jupyter lab from it. Open a browser on localhost:8888 (or if it does not work, 127.0.0.1:8888) and, when being asked for a password, type docker. From the Jupyter lab page, open the "FuncConstruct-Statistics.ipynb" notebook, and (if you wish) re-run it, or simply browse its results. Note: differently from options 1 and 2, results are not saved, but just displayed in the notebook. Using the Jupyter Notebook from your installation: this is similar to Option 3, but it can work if you have already Jupyter lab installed, with the R kernel enabled (for details see: https://github.com/IRkernel/IRkernel). The steps to follow are: Run jupyter lab (e.g., jupyter lab from the command line) and open it on a webpage. Open the FuncConstruct-Statistics.ipynb notebook. If you want to re-execute it, uncomment the install.packages() line. Re-run it (if you wish). The output If using Option 1 or 2, the results directory will contain the following files: Figures 4 and 5 as in the paper. Tables 2-9 as in the paper in various formats (csv, tex, and for Tables 3-5 also .txt). Some notes: The diagnostics (top part, up to "Fixed effects") for Tables 3-5 are shown in the .txt files only. However, these files do not report the "OR" columns that correspond to exp(Estimate). This is because the .txt file contains the statistics dump which does not include the ORs. The .csv and .tex tables report the Fixed effects as shown in the paper, including the ORs. rq1-rq2-correlation (.tex and .csv) contains the correlation analysis between RQ1 and RQ2 results as discussed in the "Threats to construct validity" (Section 6). rq3-inter-rater (.tex and .csv) contains the results of the inter-rater agreements analysis discussed in Section 3.6.