Computing integrated activities scored for programming concepts

# Computing integrated activities scored for programming concepts [https://doi.org/10.5061/dryad.k0p2ngfgj](https://doi.org/10.5061/dryad.k0p2ngfgj) **Associated journal article:** Margulieux, L. E., Parker, M. C., Uzun, G. C., & Cohen, J. D. (2023). Levels of computing concepts used in computing integration activities across disciplines. *Journal of Technology and Teacher Education, 31*(2), 167-202. Waynesville, NC USA: Society for Information Technology & Teacher Education. [https://www.learntechlib.org/p/221815/](https://www.learntechlib.org/p/221815/) Please contact Lauren Margulieux, [e-mail], for additional information about this dataset. The listed computing integrated activities were scored for which programming concepts they included. A blank cell means the activity did not include the concept, and a 1 means it did. Activities were scored binarily, either they included the concept or not. The number of times the concept is used within an activity was not counted to avoid uneven representation across shorter or longer activities. ### Initial Programming Concept Framework Integrated computing activities are often theoretically framed in a CT framework, especially for science and math activities for which national standards (e.g., the Next Generation Science Standards and Common Core) explicitly include CT. Thus, we explored CT frameworks for the programming concepts that they include (Aho, 2012; Armoni, 2016; Barr & Stephenson, 2011; Brennan & Resnick, 2012; Denner et al., 2012; Denning, 2017; Grover & Pea, 2013; Lye & Koh, 2014; Palts & Pedaste, 2020; Tang et al., 2020; Weintrop et al., 2016; Wing, 2010; Yadav et al., 2014). As Tang et al. (2020) highlight in their systematic review of CT frameworks aimed at assessment, many of the frameworks focus on problem-solving competencies rather than programming concepts. The main exception to this trend is Brennan and Resnick’s (2012) framework, which includes a category for computational concepts that lists sequences, parallelism, loops, events, conditionals, operators, and data. We used these concepts as the initial foundation for our scoring scheme. In addition, Weintrop et al.’s (2016) definition of CT for math and science integration includes specific data practices, such as creating, manipulating, and visualizing data. We transformed these practices into programming practices, such as inputting variable values, using operators to calculate variables, and various visualization concepts. To supplement the CT frameworks, we also considered how different block-based programs grouped blocks into menus. These groupings were recently analyzed in Lin and Weintrop’s (2021) landscape report of block-based programming languages. Their analysis of 36 languages grouped blocks based on concepts. The conceptual groupings included: variables/data structures, logic operators, sprite/character appearance, operators (numerical, textual, color), customize block (i.e., function), movement, sound, sense/input, interacting with the physical/virtual environment, output, debug, comment, and extension. These categories were added to our initial conceptual framework. From this theoretical basis, we began an iterative scoring of computing integration activities. Our goal was to refine the framework based on concepts that were missing and differentiate large categories into more specific concepts, such as separating the concept of loops into *for* and *while* loops. ### Revisions to Programming Concept Framework Our full, revised programming concept framework can be found in Table 1, with definitions and examples for each concept that we considered when analyzing the integrated computing activities. When considering each concept definition, it is important to differentiate the role of the programmer and of the user, which can often be the same person in integrated activities. The programmer is the person creating the program by interacting with the blocks, such as initializing a variable. The user is the person interacting with the program as it runs, such as responding to prompts to change the value of variables. More details about the evolution of the framework can be found in Margulieux, L. E., Parker, M. C., *Uzun, G. C.*, & Cohen, J. D. (2023). Levels of computing concepts used in computing integration activities across disciplines. *Journal of Technology and Teacher Education, 31*(2), 167-202. Waynesville, NC USA: Society for Information Technology & Teacher Education. [https://www.learntechlib.org/p/221815/](https://www.learntechlib.org/p/221815/) **Table 1** *Definitions of programming concepts used in integrated computing activities with examples* | ***Concepts*** | ***Definitions*** | ***Examples*** | | :---------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------- | | \*\*Algorithms \*\*– nature of the program | | | | Sequence | The order of blocks was critical to correct execution | Defining the variables before using them to calculate other variables | | Parallelism | Used multiple code sequences in the program | Multiple sprites programmed with their own behaviors | | \*\*Variables (and Data Structures) \*\*– how data is stored or handled | | | | Variable | Used a variable | Assigned a string to the variable "adjective" and used that variable later in the program | | sub-concept: +=/change | Used a += or change block to calculate a new value for a variable | Increase the value of variables to visualize the Fibonacci sequence | | sub-concept: calculated variable | Used arithmetic operators or the value of other variables to calculate the value of the variable | Subtract the value of other angles from 180 to find the value of the last angle in a triangle | | List | Used a list to assign multiple values to a variable | Used a list to provide multiple options for the variable "adjective" | | **Operators –** how data is processed or analyzed | | | | Arithmetic | Used +, -, \*, /, or other operators to calculate numerical values | Divide a wavelength in half to draw the first half of the wave | | Relational | Used <, <=, >, >=, or == | Compare the value of different fractions | | Boolean/Logical | Used and, or, not, true, false, or other logical operators | Determine the quadrant of a graph based on the x- and y-axis value | | \*\*Loops – \*\*how code is repeated | | | | For loop | Used a loop that repeated for a specific number of times entered by the user or a forever loop that repeated infinitely | A program that repeats a sequence multiple times to draw a symmetrical shape | | Sub-concept: loop index | Used an index to count or control how many times to loop | An index within a loop that counts the number of times a loop has run | | While loop | Used a loop with a termination condition | Loops until a variable becomes negative | | sub-concept: nested loop | Used a loop within another structure, like a loop | Repeat a drawing sequence within a larger repeated sequence to draw symmetrical shapes | | \*\*Conditionals – \*\*how the program makes decisions | | | | If-then | Used a statement to determine whether a condition was met to determine whether code should be executed | Compare a variable to a predetermined value to determine whether to display a message | | If-else | Used a series of statements to determine which matched a given condition | Compare a variable to a predetermined value to determine which message to display | | sub-concept: nested conditional | Used a conditional within another structure, like a loop | Repeat a conditional statement for the changing value of a variable within a loop | | **Functions –** chunks of code that are easy to reuse | | | | Define/call | Called a function defined by the user | Define a function that draws a shape | | Parameter | Included a parameter as part of the function definition | Define a function that draws and shape using a parameter for the size of the shape | | **Visualization –** how components move around the screen | | | | Movement | Moved sprites or objects to visualize a process | Characters in a story move throughout the story | | sub-concept: pen up/down | Used the pen up or pen down blocks | Using pen up and down to differentiate between different shapes | | Cartesian | Used Cartesian coordinates to determine the location of a sprite or object | Start a character in the bottom left corner by setting their location to negative x and y values | | Angles | Changed the direction of sprites or objects using angles | Draw a triangle based on the angles of vertices | | \*\*Components – \*\*audiovisual components to be visualized | | | | Multimedia | Added a multimedia component to the program such as a sprite, sound, pen, or stamp | Add two characters and a background to a digital story | | sub-concept: button | Added a button for the user to click to trigger an event | Add a button that says "Click here when ready" | | sub-concept: counter | Added a counter for the number of times an event has occurred | Count how many times a sprite has touched another sprite | | Properties | Changed the default properties | Change the color of a pen | | \*\*Output – \*\*the program communicating with the user | | | | String | Displayed a string entered by the programmer | A character is programmed to say a line | | Variable | Displayed the value of a variable, either string or numeric | The program prints the final value of a variable | | String and Variable | Displayed a string and value of a variable in the same output statement | A character says “And you picked”, + a variable selected by the user | | \*\*Input – \*\*the user communicating with the program | | | | Event | Triggered the execution of code based on the action of the user, sensor, or internal feature | A character in a story won’t talk until the user clicks them or a message is broadcasted | | Variable: String | The user entered a string into the program | A poem generator asks for a verb | | Variable: Numerical | The programmer entered a starting variable value or asks the user for input | The user picks a number | | Cleaning/transforming | Used a cleaning or transforming process on user-inputted data | Change all text to lowercase | ## Scoring Activities for Concepts Activities were scored by two of three raters for whether they included a concept. We decided to score activities for whether they included a concept rather than the number of times a concept is used. Our goal was to determine which concepts are more commonly used across computing integration activities, and scoring for the number of times a concept was used in an activity might have skewed the results based on incidental features of the activities. For example, if an activity used five variables, the conceptual knowledge required is not necessarily different than if an activity used two variables. Thus, scoring for times a concept was used was more likely to result in over-representation of the concept rather than information about which concepts were most commonly used in computing integration. The raters were three CS education researchers. Two have a Ph.D. in areas related to CS education, one from educational psychology and one from CS and have worked and published in computing education for at least five years. The other has a Master’s degree in CS and is earning a Ph.D. in instructional technology with an emphasis on CS education. Each activity was scored by at least one rater with a degree in education and at least one rater with a degree in computer science. To determine interrater reliability, we used Cohen’s Kappa. Cohen’s Kappa is appropriate to compare binary data from two raters, such as whether a concept is used in an activity. Cohen’s Kappa compares the observed agreement, in this case 96%, to the chance of agreement, calculated from the base rate of each binary option from each rater. For example, if rater one marked 80% of concepts as present in each activity on average while rater two marked 20% of concepts as present, then the chance of agreement would be low. A higher Kappa indicates higher agreement, with 0.81 to 1.00 indicating nearly absolute agreement (McHugh, 2012). For the current study, Kappa was 0.92, indicating nearly absolute agreement.