Database Control Condition

Gesture recognition technology is a popular area of research, offering many application areas, including behaviour research, Human Computer Interaction (HCI), the medical field, surveillance culture, and many more (Cheng, Yang & Liu, 2015). In this paper, I argue researchers should not only use existing recognition algorithms but build their own specialised algorithms tailored to the research questions. However, the large quantity of data needed to train a recognition algorithm is not always available, especially when researching niche phenomena. To facilitate the creation of customised recognition algorithms, in this paper I propose training and testing recognition algorithms on virtual agents, so-called avatars, a tool that has not been used for this purpose in multimodal communication research yet. To validate this approach, I provide an example use case with step-by-step instructions, using motion capture data to animate customised avatars in Unreal Engine with a varied set of lighting conditions, backgrounds, and camera angles, creating an avatar only dataset to train and test a gesture recognition algorithm. Two models are trained: a control model trained on avatars in optimal lighting conditions and neutral background, and a full model trained on avatars in both optimal and sub-optimal conditions and with cluttered backgrounds. The best performing full model reaches an accuracy of 65.9% and the best performing control model reaches an accuracy of 90.5%. The results suggest training an algorithm on artificial data is a resourceful, convenient, and effective method to customise algorithms whenever either human data is sparse or specific aspects need to be controlled for.