TracheoSpeech

The Speech with Tracheal Stoma (TracheoSpeech) Collection In this collection, we provide 24 hours of annotated speech from a patient with a severe speech impairment caused by a permanent tracheal stoma, and possibly also by neurological damage resulting from a debilitating car accident. The data was collected across 98 recording sessions of various types, amounting to 43 hours and 15 minutes of total audio (including unannotated segments). From this material, we extracted and manually annotated 11,434 short segments. Most segments correspond to individual sentences spoken by the patient, although word-level segments are also included in some cases. In addition to the extracted segments, we also provide the full session recordings. The language of both the speech and the transcriptions is Czech. We distinguish between three types of recording sessions: Artificial conversations (A): During these sessions, the patient listened to a pre-recorded dialogue and repeated the lines spoken by one of the characters. The scripts of the dialogues were generated using ChatGPT from high-level outlines written by us. The vast majority of the data was collected in this manner. The sampled segments typically correspond to individual sentences. Reading sessions (R): In these sessions, the patient read aloud from an article. She subsequently sent us both the recordings and the corresponding text, which we used to align the audio with the transcript. The sampled segments typically correspond to 1 - 3 words. Recording day (S): These sessions took place in a professional recording studio and are further divided into three subcategories. Real-world conversations (SC): The patient engaged in conversations with multiple other people. We extracted and annotated only her speech. The sampled segments correspond to either entire sentences or their parts. Singing (SS): The patient sang her favourite songs. The segments generally correspond to individual verses. Other studio recordings (SO): These sessions involved reading pre-written dialogues. The segments typically correspond to individual sentences. In the table below we provide an overview of how much data we have collected for every session type. The times are in hours:minutes format. Category Sessions Total Time Annotated Time Samples Words Artificial conversations (A) 66 36:05 19:18 8,099 30,628 Reading sessions (R) 20 4:08 2:54 2,581 4,598 Real-world conversations (SC) 2 1:31 0:28 300 1,044 Singing (SS) 7 0:34 0:32 263 1,245 Other studio recordings (SO) 3 0:55 0:42 191 869 Total 98 43:15 23:57 11,434 38,384 Structure of the dataset The TracheoSpeech.zip archive contains the following files and directories: README.md: This file. It provides an overview of the dataset contents, structure, and usage. metadata.csv: A CSV file containing metadata for each individual sample. Each row corresponds to one audio segment. The features included are described in detail in the *Features* section below. samples: A directory containing a separate audio file for each annotated segment — 11,434 files in total, amounting to 23 hours and 57 minutes of speech. sessions: A directory containing the complete recordings of all 98 sessions, including both annotated and unannotated segments—43 hours and 15 minutes of audio in total. We include this data as it may come in handy for merging short consecutive segments (for instance, machine learning models might benefit from longer context) or other tasks that make use of the unlabelled parts. Features The metadata.csv file lists all annotated audio samples. For each sample, it provides the following features: sample_audio: The filename of the corresponding audio segment, located in the samples/ directory. transcription: The manual transcription of the sample in Czech. session_type: A session type code indicating the kind of session the sample belongs to: A = Artificial Conversation, R = Reading, SC = Real-world Conversation, SS = Singing, SO = Other Studio Recording. session The identifier of the session the sample comes from. Also used as the filename of the full session recording in the sessions/ directory. start: The onset time (in milliseconds) of the sample within the full session recording. end: The offset time (in milliseconds) of the sample within the full session recording. split: This column indicates the data split in which we used this sample in our baseline method (values: train, val, test). Each sample’s audio can therefore be accessed either directly or by slicing from the full session recording: load_audio("samples\"+sample_audio) # or load_audio("sessions\"+session+".mp3")[start:end] Note that the latter option will work when processing the files with precise tools like Python pydub (recommended). The coordinates might not work for standard media players as they often introduce additional delays to the MP3 files. Baseline Method This data was collected for the purpose of the development of an ASR system. The thesis describing our approach and results as well as the implementation itself can be found in this GitHub repository. Note that the data used for our own experiments is slightly different from the data provided here because we removed a few hundred samples that referenced personal friends of our patient from the public dataset. Acknowledgement This work was done as a bachelor's thesis in cooperation with the Institute of Computational Perception at Johannes Kepler University Linz and under the supervision of Florian Schmid, Dipl.-Ing.