Data of an exploratory user study comparing interaction techniques in different Virtual Reality scenarios

Motivation

This data was collected during a user study where multiple interaction techniques were compared in different Virtual Reality scenarios. Twelve interaction techniques were evaluated in selection and manipulation tasks. Not all evaluated techniques are capable of both selection and manipulation. Therefore, we tested eight techniques in the manipulation sub-study and ten in the selection sub-study. The twelve tested interaction techniques were Bimanual Fishing Reel, Expand, Flashlight, Go-Go + PRISM, Head Based Selection, IntenSelect, Scaled HOMER, Scaled HOMER + Scale, Simple Virtual Hand, Spindle, Scaled Scrolling World in Miniature + Scale, and Crank Handle (Controller). For details on the techniques, we refer to S3DIT (10.5281/zenodo.4292312), a web application for finding suitable interaction techniques. The Hardware Setup consisted of an HTC Vive Pro with the corresponding controller and an Alienware 17 R4 (Intel i7-7700HQ, NVIDIA GTX 1070, 16 GB Ram). Unity was used to implement the test environment (10.5281/zenodo.4172112).

Tasks

In the selection task, a red ball must be selected (see images/task_selection_1.png), whereupon a structure of one or more balls appears, with the target colored green (see images/ task_selection_2.png). Regardless of the interaction technique used, the ball hit by the selection tool is highlighted with a yellow border to increase the comparability of the techniques. The time measurement started as soon as the red ball was selected and stopped as soon as the correct ball was selected.

In the manipulation sub-study, the task was to position, rotate and scale a cube with six different colored areas in such a way that it fits as closely as possible into a transparent representation of the cube (see images/task_manipulation.png). The position may differ 10 cm, the rotation 20 degrees, and the size of the cube 10 cm from the values of the transparent cube. The solution of the task must be actively initiated via the grip button of the Vive Controller. If an attempt was made to solve the task although the cube was not yet within the tolerances, nothing happened. After tests in a preliminary study, the cube was extended with two bars to provide additional spatial indicators. Besides, there were cones of different colors centrally under the objects, which should help to better estimate the distance between the objects. The time measurement started as soon as the object was gripped and stopped as soon as the solution was accepted.

Regardless of the task type, the test persons had 30 seconds to complete the task successfully. If this was not possible, this counted as a failed attempt, the current task structure disappeared and the next task was displayed. If a task was solved correctly, a small animation with stars was triggered, accompanied by an acoustic signal and the next task was displayed. In both task types, the objects were created relative to the position of the head, so that each test person could view the objects from the same perspective.

For each combination of independent variables (without technique), two tasks were randomly generated for manipulation and three for selection and presented in random order. This resulted in 81 tasks for selection and 90 tasks for manipulation. For further details on the tasks used, we refer to the repository of the testbed application (10.5281/zenodo.4172112). The tasks are run through randomly, with some restrictions, as not every technique supports every task. Some techniques only allow interaction at short distances or do not support the scaling of objects. For this reason, dummy tasks have been generated to replace tasks that are not supported by the techniques. These always come at the end to allow better comparability for the supported tasks, because then the dummy tasks do not act as training. Thus, in selection and manipulation, short distance tasks came first for each technique, and in manipulation, tasks that allow positioning and rotation came first.

Due to the many techniques that were compared, no Within-Subject Design was possible, and not every technique was tested by every test person. Also, a Within-Subject Design was not possible, because a much higher number of test persons would have been necessary. For these reasons, an intermediate design was chosen. Since the tasks could be carried out more quickly during selection, five techniques per test subject were tested here and three techniques per test subject during manipulation. The techniques were randomly assigned but evenly distributed.

Dependent and independent variables

The independent variables for the selection task were distance (0.6 m, 3 m, and 6 m), object size (15 cm, 10 cm, and 5 cm), and object density (single, 10, or 5 cm between objects). For the manipulation task, the independent variables were distance (0.6 m, 3 m, and 6 m), task type (positioning, rotation, scaling, positioning + rotation or positioning + rotation + scaling), and degree of manipulation (low, medium or high). The degree of manipulation influences the number of axes on which the object must be moved and to which extent the manipulation had to be performed. As none of the techniques allows asymmetric scaling, the enlargement or reduction of objects always took place on all axes.

Speed and precision were recorded as objective measurements. The speed is determined by the time required to perform the task. In the case of selection, the precision results from the number of failures, i.e. how often the wrong object or no object was selected until the correct object was selected. In the case of manipulation, the difference to the target object for the position, rotation, and size of the object is recorded individually to determine the achieved precision.

The System Usability Scale was used to determine usability. Besides, custom questions were asked to capture the additional factors naturalness, fun, precision, speed, and motion sickness. The NASA Task Load Index (TLX) was used to record the subjective effort involved in performing the tasks. The test persons could make additional comments via a comment field.

After all techniques had been tested by a test person, they were also asked to sort the techniques according to their own preferences. It is important to note that the test subjects did not always necessarily use techniques that could perform the same tasks. When evaluating the techniques, it was therefore important to independently weigh up various factors such as usability and expressiveness of the techniques.

Test Procedure

Initially, the test person filled in a personal questionnaire (data/demographic_data_[eng|ger].csv), asking for demographic data (age, handiness, gender, wearer of glasses and occupational field) and the level of experience with certain devices and input methods (computer, mouse, touch screen, body recognition, finger recognition, eye tracking and 3D controller), as well as certain applications (3D applications, 3D computer games and VR/AR applications). A further document, which had to be read by the test persons on site, described the task to be performed in detail and explained the necessary buttons of the controllers. If the test person had no further questions, he was led into the test area and the VR headset was put on. Even if a technique was tested that works with one hand, the test person was given both controllers, which also had to be held in the hands throughout the entire process. This was followed by a short introduction to the virtual environment. Then the training phase of the first technique began. Here, depending on the complexity of the technique, 1-4 explanatory texts were read out, which were created using Google Text-to-Speech. The speed at which the explanatory texts followed each other could be controlled by the study director. The test subjects then had five minutes to familiarize themselves with the technique in previously generated sample tasks. These sample tasks were similar to the actual tasks. However, the test persons could also say that they wanted to start with the tasks before the time was up. Then the task phase began.  For the completed tasks, the required time and failed attempts or the achieved precision was recorded (data/measurements_manipulation_.csv and data/measurements_selection_.csv). Once all the tasks were completed, the test person removed the headset and had to answer the System Usability Scale and the custom questions on a PC (data/system_usability_scale_and_custom_questions_[eng|ger].csv). The questionnaires were implemented using Google Forms. The NASA Task Load Index was filled out in paper form (data/nasa_tlx_[eng|ger].csv). The test person was offered a break and then he continued with the next technique. After all techniques were tested the final questionnaire for ranking the techniques was filled in (data/ranking_[eng|ger].csv).

Files

The study was done in German. The questions and results can be found in the files ending with “ger”. The English translation can be found in the files ending with “eng”. The files containing the results of the questionnaires include the asked questions and don’t need further explanations. In the following, only the columns of the file measurements_manipulation and measurements_selection are explained:

UserId: A number identifying the test person. The same number in different files identifies the same test person.

InteractionTechnique: Name of the interaction technique used.

TaskID: Internal id of the task.

Type: Tasks that needed to be executed.

TaskObject: The used task object. For the selection task, this indicates the size of the object.

Distance: The distance between the test person and the task object(s).

NumberOfObjects: The number of task objects.

MinDensity: The minimum distance between two task objects.

NeededDoFs: The number of axes on which the object needs to be manipulated  (1=x, 2=x, and y, 3= x,y, and z).

ManipulationAmount: The amount the object needs to be changed in position (1 = 1/3 m, 2 = 2/3 m, 3 = 1 m), rotation (1 = 45-90°, 2 = 90-135°, 3 = 135-180°) or scale (1 =  factor 1.25-1.5, 2 = factor 1.5-1.75, 3 = factor 1.75-2).

Success: Depicts whether the test person was able to successfully finish the task within the time limit.

Time: The needed time.

Misses: Number of wrong selections until the correct object was selected

PositionDifference: The position difference between the task object and the target object at the end of the task.

RotationDifference: The rotation difference between the task object and the target object at the end of the task.

ScaleDifference: The scale difference between the task object and the target object at the end of the task.