Janneke van der Steen - PhD Project data for study 1

<B>Title</B><BR><BR>
Supporting Teachers in Improving Formative Decision-Making: Design Principles for Formative Assessment Plans<BR><BR>
<B>Abstract</B><BR><BR>
Formative assessment is considered as one of the most effective interventions to support teacher decision-making and improve education and student learning. However, formative assessment does not always meet these expectations. In order to be effective, formative assessment activities should be consciously and coherently planned aligned with other aspects of the curriculum and the decisions teachers wish to make based on these activities. While there is sufficient support for teachers to design formative assessment activities, no guidelines exist to help them tie these different activities together in an effective way. To support teachers in designing formative assessment plans informing formative decision-making, this study focused on the creation of a set of design principles. These design principles for formative assessment plans were formulated based on expert interviews and subsequently evaluated by future users. The result is a set of eight design principles that can be used and validated in educational practice.<BR><BR>
<B>Description of the data included</B><BR><BR>
1. <B>Four lists of clustered critical aspects for a formative assessment plan mentioned by the participants during the expert interviews. </B><BR>In total, twenty participants were experts from both research and practice, all involved with formative assessment. Table 1 shows the combination of experts in each expert group:<BR><BR>
<I>Table 1. Participant expert interviews</I><BR>
<B> Groups - Participants</B><BR>
Group 1 - Three educational researchers, one teacher-educator, and one school policy maker<BR>
Group 2 - Two educational reseachers, three teachers from one school for secondary education<BR>
Group 3 - Nine teachers from two schools for secondary education, one teacher-educator<BR><BR>

The purpose of these expert interviews was to reach agreement among the participants of each group about what they thought were critical aspects a formative assessment plan should have to be effective. These critical aspects will be clustered, first within groups and then across groups, and used in a later stage as a starting point to formulate design principles. To promote consensus, the interviews were organized as group decision rooms where the discussion was supported by a digital group support system (Fjermestad and Hiltz, 2000; Pyrko et al., 2019). This support system offers participants the possibility to answer questions individually and digitally through a device followed by a group discussion about how to cluster all given answers. By clustering their own answers during the interview within the group, participants were directly involved in the first phase of data-analysis. All subsequent steps that were taken in the expert interviews are presented in Table 2:<BR><BR>
<I>Table 2. Activities and questions expert interviews</I><BR>
<B> Activity - Question</B><BR>
1. Participants answer individually - "Can you name three critical aspects of a formative assessment plan?"<BR>
2. Group discussion - "How can we cluster the given aspects? What name should the different clusters have?"<BR>
3. Participants answer individually - "Which two critical aspects a formative assessment plan should have are still missing in the composed list?"<BR>
4. Group discussion - "Can we add these extra aspects to existing clusters or do we need to create new ones?"<BR>
5. Participants answer individually - "If you still think that there are critical aspects missing in the composed list, can you please add them now?"<BR>
6. Group discussion - "Can we add these extra aspects to existing clusters or do we need to create new ones?"<BR>
7. Participants answer individually - "How would you arrange all clustered critical aspects that are the result of this expert interview, in order of importance?"<BR><BR>

2. <B>Anonymized Transcripts of four group interviews with future users originating from four schools for secondary education set up to evaluate the draft version of the design principles for formative assessment plans. </B><BR>Each group consisted of five to eight teachers from the same school. In two cases, a school leader also joined the interview, see Table 3:<BR><BR>
<I>Table 3. Participants group interview</I>s<BR>
<B>Groups - Participants</B><BR>
School 1 - Five teachers<BR>
School 2 - Seven teachers and two school leaders<BR>
School 3 - Five teachers and two school leaders<BR>
School 4 - Six teachers and one school policy maker<BR><BR>

The teachers and school leaders were questioned about recommendations regarding transparency, usability, completeness, and suitability of the design principles for school practice. The participants had received the design principles in advance. First, the participants were asked to write down all recommendations they could think of to improve the design principles. Secondly, they were asked to decide what facet of the design principles would improve if this recommendation was followed. Facets they could choose from were transparency, usability, completeness, or suitability. Subsequently, they were asked to give explanations of their recommendations and the improvements they would expect the transcripts consist of this discussion of the explanations and recommendations.