RAGE_Pilot2_SPL-ITAlert

General description:

The data are gathered during an evaluation of the first prototypical version of the game IT Alert within the RAGE project.

This evaluation investigated the quality of the ecologically valid educational setting with the game (i.e. game & debriefing) for acquiring team skills (communication and collaboration). The quality of the game was examined by analysing participants’ opinions on usability, studyability, and perceived usefulness of the game for learning. In addition, more objective measurement of participants’ learning progress was gathered via in-game traces (i.e., in-game scores on two variables regarded important in IT team functioning while maintaining the integrity of a networked system). The main research question targeted in this evaluation was: Does the educational setting with the game (i.e., game & debriefing) lead to learning progress, and by which effort and satisfaction of the learner (task load, motivation)?

In total, 47 students installed and started IT Alert, but only 24 students truly participated in the most relevant part of the evaluation, completing two game sessions and the evaluation questionnaires.

According to the results, performance gains occurred from the educational setting with the game.

Topic
ACM CSS 2012: Applied computing~Interactive learning environments   • Human-centered computing~Usability testing 
PsycINFO Classification: 3500 Educational Psychology - 3530 Curriculum & Programs & Teaching Methods; 3550 Academic Learning & Achievement

Name entitites
Organizational information: Open University of the Netherlands
Geographical information: The Netherlands
Time information: February 2018 – November 2018

Types: SPSS-file (version 24)

RAGCS target group: demand side – end-users - adults- vocational school

Evaluation dimensions
Evaluation object: IT Alert game
Methodology/design: The research focuses on the combined role of game and debriefing [three variants] (using game-traces from session 1 [as pre-test] and session 2 [as post-test]) on the development of the individual in team functioning (looking at individual performance & team performance and considering collaboration and communication as team skills).
Evaluation variables: Communication skills, Self-perceived communication knowledge, Communicatin attitude, Usability, Studyability, Task load, Motivation

Instruments: Communication skills (in-game-traces); Self-perceived communication knowledge (ICKA, Wilkins et al., 2015); Communication attitude (CSAS, Laurence et al., 2012); Usability (SUS, Brooke, 1996); Studyability (Nadolski & Hummel, 2017); Task load (NASA-TLX, Hart & Staveland, 1988); Motivation (IMI, Ryan & Deci, 2000)

Knowledge/skill elements
RAGCS skills: skills - cognitive skills – applying; skills - affective skills - interpersonal

ESCO skills: coordinate communication within a team (http://data.europa.eu/esco/skill/cd85312f-0e0b-460b-addd-57d6df800595)

Relationships: D8.4 Second RAGE Evaluation Report