LSE-HEP-UVigo Dataset

LSE-HEP-UVIGO is a LSE dataset focused on hospital emergency triage interactions.

The dataset was created to support the development of an application enabling communication between deaf patients and hospital staff in emergency settings when no other interpreting resources are available. It includes the standardized questions of the triage protocol, which hospital staff must ask to assess time-sensitive diseases and injury conditions, together with a set of predefined answers and a set of free-form answers.

In total, the corpus contains the following data:

73 questions (LSE RGB videos) signed by a deaf person (male), 

890 predefined answers to those questions (LSE RGB videos), signed by 4 deaf persons (3 female, 1 male),

3588 repetitions of the 890 answers with slight alterations in the sequence of signs (MediaPipe Holistic keypoints), signed by 85 deaf persons,

1486 free responses to the 73 questions (MediaPipe Holistic keypoints), signed by the same 85 deaf persons.

The released annotations include ID-glosses, pseudo-glosses, and Spanish translations for the predefined answers, and Spanish translations only for the free-form answers. MediaPipe Holistic keypoints are provided for the entire dataset, while RGB videos are released only for the subset of questions and predefined answers signed by the five deaf participants hired within the project. The 85 participants were recruited through deaf associations and did not grant permission for their images to be shared. 

Due to the rich annotations, this dataset is prepared for training Continuous Sign Language Recognition and Sign Language Translation, as well as fingerspelling recognition and sign spotting.

Distribution files:

LSE-HEP-UVIGO_metadata.xlsx: contains 4 sheets with the next subsets: QUESTION_Table, ANSWER_predefined_Table, Donated_copy_sentence_Table and Donated_free_answer_Table. The excel file contains the identification of every mp4 and pkl file, as well as the ID-gloss, pseudogloss sequences and Spanish translations.

features_mediapipe.tar.gz: contains three folders with the following data: QUESTIONS,  ANSWERS_PREDEFINED, ANSWERS_DONATED (contain the features of both type of donations: predefined answers and free answers). All files are MediaPipe Holistic keypoints in pkl format.

videos_RGB.tar.gz: contains two folders with the following data: QUESTIONS_RGB, PREDEFINED_ANSWERS_RGB.

train_val_test_split.csv: contains the list of file_ids with the split it belongs to (train 71,3%, val 13,1%, test 15,6%). This split is prepared for signer independent evaluation.