Data for: How do we measure and increase systems thinking? Comparing self-reported and performative metrics in response to building causal loop models

Systems thinking is a mindset and skill essential to understanding and taking effective action to address complex challenges. However, increasing and measuring systems thinking is difficult. This data set includes multiple approaches to measuring systems thinking. It includes 3 self-reported measures of a systems thinking mindset, and 2 performative measures of systems thinking skills (metrics derived from causal model diagrams of a system that participants created, and accuracy of responses to simple trophic cascade scenarios). The data allow for a comparison between different measurement approaches. They also allow for the evaluation of whether systems thinking is specific to disciplinary domains (ecological vs economic) or generalizes across domains.  Finally, the data provides a preliminary assessment of whether building causal models increased systems thinking.

Methods
Participants (N = 185) were workers from Amazon Mechanical Turk (MTurk) and located in the United States. Participants were paid $5.00 (roughly $10/hour).

Our survey used a mixed-methods counterbalanced design, resulting in both within-and between-subjects measures described in the table below. 

The survey was administered using Qualtrics and was entirely online. After providing informed consent, all participants completed pre-measures (the exact measures varied based on random assignment). All participants then watched a 5-minute instructional video created by the researchers and available at https://vimeo.com/479005460. The video explained and provided an example of how to use Mental Modeler. It did not, however, use the term model or system or discuss or explain anything about systems thinking.

Participants then read a written description of the ecological system. This narrative described how the foxes were dependent on the rabbits as a food source, and that the foxes reduced the number of rabbits by eating them, how the rabbits were dependent on the grass, how the grass was fertilized by the droppings of both species, etc. In addition to including the causal relationships just described, the narrative also included several non-causal factors. For example, it was stated that the landscape contained rocks. The description was written to define an unambiguously correct set of causal relationships that would be present in the model. A link then took participants to the Mental Modeler software, where they were asked to create a mental model of the scenario described to them. Participants were instructed to download the file produced by the software, as well as take a screenshot. Both were uploaded into the online survey. Participants then continued with the survey to complete the appropriate post-measures, given their random assignment. The measures are described below.

Measure

Description
Role in Study
Sample item

Systems Thinking Scale (Randal & Stroink, 2018)
Measures self-reported dispositional systems thinking
Measured pre-MM exercise
Q: “The Earth, including all its inhabitants, is a living system.”
A: 1 = “Strongly Disagree” to 7 = “Strongly Agree”

Ecological Knowledge (developed for this study)
Measures self-reported familiarity with ecological concepts
Measured pre-MM exercise, for use as covariate
Q: “I have taken a biology or environmental science course in college or high school that provided me with an understanding of basic ecological concepts.”
A: “Yes” or “No”

Local Systems Perception Scale (Petersen et al., 2018)
Self-report measure of the extent to which participants see themselves as part of local ecological, economic, and social systems
Measured pre- and post-MM exercise, to evaluate whether MM exercise changed answers
Q: “I think of the place I live as a social system composed of interrelated parts.”
A: 1 = “Strongly Disagree” to 5 = “Strongly Agree”

Perceptions of Causal Connection (Petersen et al., 2018)
Measures participants’ self-reported tendency to see causal connections across different spatial and temporal scales
Measured pre- and post-MM exercise, to evaluate whether MM exercise changed answers
Q: “My local actions affect things that happen at a global scale.”
A: 1 = “Strongly Disagree” to 5 = “Strongly Agree”

Ripple Effect scenario
(based on Maddux & Yuki, 2006)
• Two ecological scenarios (goldfish, cat)
•One economic scenario (bookstore)

Description of a trophic cascade scenario that presents both relevant and irrelevant factors

Seen either before or after MM, counter-balanced
“A neighborhood borders a nature preserve …[that] contains a food web composed of sedge plants, insects, mice, lizards and hawks.  … [a free-roaming cat] catches and kills mice but does not hunt either lizards or birds.”

Ripple Effect impact questions
Performative assessment of the tendency to accurately identify chains of causality rippling out from a single event
Performative measure of systems thinking. Measured either before or after MM, counter-balanced, to evaluate whether MM exercise changed answers
Q: “Use the choices below to indicate the degree of impact of this introduction on the reservoir …
1. the goldfish”
A: -2 = “Decrease substantially” to 2 = “Increase substantially”

Ripple Effect responsibility questions
Performative assessment of perceived responsibility for changes to a system

Performative measure of systems thinking. Measured either before or after MM, counter-balanced, to evaluate whether MM exercise changed answers

Q: “To what extent are the following responsible for the impacts of the goldfish on the reservoir?
1. Ted’s parents”
A: 1 = “Not at all responsible” to 3 = “Very responsible”

Mental Modeler metrics
Numbers of each model feature
Performative measure of model-building
Descriptive metrics of participants’ models
Number of positive arrows

Mental Modeler accuracy metrics
Performative measure of accuracy of each variable for each model

Performative measure of systems thinking. Completed between the two ecological ripple questions
Number of correct positive arrows