Analyzing Deaf and Hard-of-Hearing Users’ Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input

This dataset is being shared to accompany the following CHI'2022 publication: Abraham Glasser, Matthew Watkins, Kira Hart, Sooyeon Lee, and Matt Huenerfauth. 2022. Analyzing Deaf and Hard-of-Hearing Users’ Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input. In CHI Conference on Human Factors in Computing Systems (CHI ’22), April 29-May 5, 2022, New Orleans, LA, USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3491102.3501987 We release our video-recordings and accompanying annotation, to support future Human-Computer Interaction (HCI) research on the behavior of Deaf and Hard-of-Hearing (DHH) users of personal assistant systems, as well as serving as potential data for sign-language recognition researchers who are training artificial-intelligence models for their software. These video recordings consist of our DHH study participants issuing commands to a personal assistant system. Also available is a CSV file with our dataset annotation. Access: To access this dataset, you should become an Authorized User of Databrary: https://databrary.org/about/agreement.html After becoming an authorized user of Databrary, please contact Matt Huenerfauth if you have difficulty accessing this volume. Dataset description: There are 21 "session" folders -- 1 for each participant in the user study. Each folder is named "P01", "P02", and so on, until "P21". In each respective folder, is a .mp4 file named "P01", "P02", and so on. These files are the video recordings from the user study as described inside the CHI 2022 paper (above). There are also two supplementary materials. One is a .csv file named "demographics_for_sharing_chi2022". This file is some basic demographics for the 21 participants in this study. The columns inside this .csv file and description of their contents are as follows: ID - Arbitrary consecutive integers to label a row Gender - Gender of the participant Birth Year - Birth year of the participant How would you describe yourself? [Deaf/deaf/Hard of Hearing/Hearing] - Participant answer to this question. At what age did you become DHH? - Participant answer to this question. At what age did you begin to learn ASL? - Participant answer to this question. Are your parents DHH? - Participant answer to this question. Did your parents use ASL at home? - Participant answer to this question. In elementary school, did you use ASL? - Participant answer to this question. What language do you use at home? [0= 100% English 0% ASL, 10= 0% English 100% ASL] - Participant answer to this question. What language do you use at work/school? [0= 100% English 0% ASL, 10= 0% English 100% ASL] - Participant answer to this question. What language do you use with friends/family? [0= 100% English 0% ASL, 10= 0% English 100% ASL] - Participant answer to this question. The second supplementary file is another .csv file named "data_annotations_chi2022". Inside this file is the data annotation for the 21 videos, as described in the CHI 2022 paper. The columns inside this .csv file and their contents are as follows: Participant ID - ID for the participant Video Filename - Name of the .mp4 file associated with the annotation Command number within Video ID - Number of command inside the video -- each video has multiple commands Wake up Method (only on the first of each participants) - Which wake-up method was employed by the participant (see CHI 2022 paper) Command in English - Transcription of the ASL command in English Error Type (If it happens) - Error type, which will be one of the following: ----Silence (Alexa ignored the command) ----Confusion (Alexa heard but didn't understand the command, Alexa says something like ""I don't know that"") ----Suggested (#2 but Alexa suggested something at the end) ----Undesired (Alexa understood the command but didn't give the desired result) ----Failure (Alexa crashed, hardware error, software error (e.g. captions stuck), ...) ----Question (Alexa understood but missed key information thus asking the participant for confirmation) ----N/A (No error, Alexa responded as expected) How participant followed up on the error - Follow-up type, which will be one of the following: ----Repeated (Self-explanatory -- same signing and wording) ----Reworded (Self-explanatory -- changed signing or wording, including only changing the greeting) ----Ignored (Ignored, moved on) ----Question (The participant asked either Alexa or the researcher about the error they're seeing -- the participant basically saying "what can i do?" or "what do i do?") ----Played Along (When Alexa suggested something, sometimes they'll accept the suggestion even though it wasn't what they were expected. Otherwise, the error type was #4 but they accepted and asked about that result) ----N/A Please see the CHI 2022 paper for further information and explanation on this dataset that is being shared with it. Reporting Bugs or Errors: Please contact Matt Huenerfauth to report any bugs or errors that you identify in the corpus. We appreciate your help in improving the quality of the corpus over time by identifying any errors.