USE-ME Empirical evaluation pilot study

We report on the pilot assessment of the feasibility of the USE-ME tool with master students in computer science which were involved into a DSL course. In total there were four groups consisted of two or three participants which were developing the following DSLs:

DSL Spreadsheets - a DSL transforms an activity graph into a Gantt chart. The target users of this DSL are to be project managers.

Gestures Kinect - a DSL which supports specification of communication between navy users using a Kinect device. The target users of this DSL are to be a navy operators.

Peddy Paper - a DSL which creates several instances of personalised paddy papers. The target users are paddy paper builders.

Smart House - a DSL which supports a design of the elements and operations that the house can have. It is meant to be used by house owners. 

The evaluation consisted of three learning session. The first one took place after four weeks of the DSL development. Students were introduced to the usability evaluation during a 2h theoretical lecture. For next 2h were introduced with the USE-ME tool and were guided to perform installation and set up working environment. Also, the students were given a participation questionnaire to fill in and describe a purpose of their DSL. In the end of a session, they were given the background questionnaire to fill till the following session which was happening 1 week after. Following two weekly 4h sessions were consisting of USE-ME hands on. The students were introduced to the modelling activities followed by Visualino example. For each activity, the same student from the group was using the tool to create USE-ME models for their DSL, while other students from the group were helping in deciding what would be a right specification. Finally, students were asked to try to finish the learned models and deliver them as a part of DSL course. After delivery, the students which were using the tool to model were asked to fill in a feedback questionnaire. 

The evaluation deliveries related to each DSL (project reports, USE-ME models), and ones related to participants (background and feedback questionnaire), as well as evaluation results, are attached in this data set.