Using a "sledgehammer" approach to increase systems thinking with a brief manipulation

Systems thinking is a skill that is essential to understanding and taking effective action on complex challenges such as climate change. This research evaluated whether systems thinking could be increased with a brief intervention. Participants (N = 678) recruited from Amazon Mechanical Turk all completed the Systems Thinking Scale (Randel & Stroink, 2018), which was used as a covariate. Participants were then randomly assigned to one of four conditions. Some participants (n = 165) watched an entertaining 5-minute video describing systems thinking with a real-life example (Cats in Borneo, https://www.youtube.com/watch?v=17BP9n6g1F0). Others (n = 174) watched this video, read a definition of systems thinking, and were asked to engage in systems thinking while completing a survey. This was designed to be a "sledgehammer" condition, in which we made our manipulation as heavy-handed as possible. A third (control) condition (n = 167) watched a video about how to fold a fitted sheet. A final control condition (n = 172) watched no video. All participants completed a survey that included nine different measures that capture different aspects of systems thinking. Despite large sample sizes and multiple operationalizations of systems thinking, support for the efficacy of our brief intervention was weak at best. Those who watched the systems thinking video scored significantly higher on one self-report measure of the extent to which people perceived themselves to be part of local social, economic, and ecological systems. Watching the systems thinking video also marginally increased the accuracy with which people correctly identified positive and negative feedback loops, and the extent to which participants saw a shot in a pool game impacting the outcome of the game. However, the control condition performed significantly better than the two systems thinking video conditions on a stock/flow identification task, and all other measures showed no differences by condition. We conclude that increasing systems thinking with a brief manipulation, even one that defines systems thinking and begs participants to engage in systems thinking, is not very effective.

Methods
Participants were recruited via Amazon Mechanical Turk. All participants were adults living in the United States. 

We gave 10 different measures that capture some aspect of systems thinking:

The Systems Thinking Scale (Randle & Stroink, 2018):

This 15-item self-report scale measures someone's dispositional tendency to engage in systems thinking. Negatively worded items were recoded, and all items were averaged together. Higher scores = more systems thinking. This trait measure was given at the start of the study and was used as a covariate.

The Murder Scenario (Choi et al., 2007):

Participants read a brief description of a murder case and indicated which of 96 possible facts were irrelevant to the case. We recoded the items such that 1 = relevant, 0 = irrelevant. The recoded items were summed together. Choosing more items indicates more holistic thinking about causality.

Ripple Effect Question: Driver Scenario:

Based on measures developed by Maddux & Yuki (2006). Participants read about a minor traffic accident. They estimated the level of responsibility that different actors in the scenario have for various outcomes, and estimated the impact that the accident had. The responsibility and impact items were each averaged together. Seeing wider rings of responsibility and impact indicates more holistic thinking about causality

Ripple Effect Question: Pool Game. (Maddux & Yuki, 2006):

Participants saw a picture of someone taking a shot in a pool game. They indicated the extent to which this shot would affect the next several shots and the outcome of the game. Seeing more impact farther out in time indicates systems thinking.

Ripple Effect Question: Executive

Based on measures developed by Maddux & Yuki (2006). Participants read a scenario about a corporate executive who must make difficult personnel decisions to address financial difficulties. They estimated the number of people affected by the decision. They also rated the level of responsibility the executive had for various outcomes. Seeing wider rings of responsibility and impact indicates more holistic thinking about causality.

Pharmacy Scenario:

Based on a measure developed by Chiu et al. (2000), where participants read about a situation in which a pharmacy technician gave children in a hospital the wrong medication, causing many to become ill. Participants indicated the extent to which characteristics of the technician (dispositional attributions) or the situation (situational attributions) were important causes of the event. More situational attributions indicate more holistic thinking about causality.

Plot identification. (Rottman et al., 2012):

A verbal description of a causal system was shown above four plots. One plot accurately depicted the verbal description. Participants were asked to choose the plot that best depicted the causal system described. We totaled the number of correct answers.

Picture Mapping. (Vendetti et al., 2014):

Participants were presented with ten sets of two pictures: a highlighted object in the first picture had both ‘relational’ and ‘object’ matches in the second picture. Participants were asked to choose the object in the second picture "that goes with the highlighted item in the first picture." Choosing the relational match indicates more relational reasoning. We totaled the number of relational answers.

Stocks and Flows. (developed for this study):

Participants read brief definitions of "stocks" and "flows". They were then presented with 14 items (e.g., water in a reservoir, deaths per year) and asked to indicate whether each item was a stock or a flow. We totaled the number of correct answers.

Feedback Loops. (developed for this study):

Participants read a definition of feedback as well as an example of positive feedback and negative feedback. They were then presented with eight phenomena (e.g., When a herd animal is alarmed and startles, this causes others to startle). Participants were asked to indicate whether each was an example of positive feedback, negative feedback, or neither. We totaled the number of correct answers.

Participants completed the Systems Thinking Scale, then were randomly assigned to one of four conditions. Some participants (n = 165) watched an entertaining 5-minute video describing systems thinking with a real-life example (Cats in Borneo, https://www.youtube.com/watch?v=17BP9n6g1F0). Others (n = 174) watched this video, read a definition of systems thinking, and were asked to engage in systems thinking while completing a survey. This was designed to be a "sledgehammer" condition, in which we made our manipulation as heavy-handed as possible. A third (control) condition (n = 167) watched a video about how to fold a fitted sheet. A final control condition (n = 172) watched no video. All participants then completed the other nine measures listed above.

References

Chiu CY, Morris MW, Hong YY, Menon T. 2000. Motivated cultural cognition: the impact of implicit cultural theories on dispositional attribution varies as a function of need for closure. Journal of Personality and Social Psychology 78: 247–259.

Choi I, Koo M, Choi JA. 2007. Individual differences in analytic versus holistic thinking. Personality and Social Psychology Bulletin 33: 691–705.

Maddux WW, Yuki M. 2006. The “ripple effect”: cultural differences in perceptions of the consequences of events. Personality and Social Psychology Bulletin 32: 669–683.

Randle, J. M., & Stroink, M. L. (2018). The development and initial validation of the paradigm of systems thinking. Systems Research and Behavioral Science, 35(6), 645-657.

Rottman BM, Gentner D, Goldwater MB. 2012. Causal systems categories: differences in novice and expert categorization of causal phenomena. Cognitive science 36: 919–932.

Vendetti MS, Wu A, Holyoak KJ. 2014. Far-out thinking generating solutions to distant analogies promotes relational thinking. Psychological Science 25: 928–933.