Supplemental Material for "Which Experimental Design is Better Suited for VQA Tasks? - Eye Tracking Study on Cognitive Load, Performance, and Gaze Allocations"

We investigated the effect of stimulus-question ordering and modality in which the question is presented of a user study with visual question answering (VQA) tasks. In an eye-tracking user study (N=13), we tested 5 conditions within-subjects. The conditions were counter-balanced to account for order effects. We collected participants' answers to the VQA tasks, responses to the NASA TLX questionnaire after each completed condition, and gaze data was recorded only during exposure to the image stimulus. We provide the data and scripts used for statistical analysis, the files used for the exploratory analysis in WebVETA, the image stimuli used per condition and training as well as the VQA tasks related to the images. The images and questions used in the user study is a subset of the GQA dataset (Hudson and Manning, 2019). For more information see: https://cs.stanford.edu/people/dorarad/gqa/index.html The mean fixation duration, hit-any-AOI rate and scan paths were generated using gazealytics (https://www2.visus.uni-stuttgart.de/gazealytics/). The Hit-Any-AOI-rate and mean fixation duration was calculated per person per image stimulus.