ontocord/youtube_commons_with_cleanups

---
license: cc-by-2.0
---
This is a copy of a subset of the youtube_commons dataset which includes transcripts from CC-BY-* youtube videos. We have run the trancript through a cleanup model to cleanup the text and add punctuations. This is by no means perfect, and there are still lots of grammatical issues.
We then combined all transcripts from the same video into one data time, separated by <|endoftext|>. When possible, we included English text first.

Where there was no English text, we machine translated the first few words of the text into English and prepended the English text.

We added some basic formatting such as "Title:", and "Transcript:".

Also we ran the English text through a toxicity model and recorded the results.

Lastly we used an edu scorer to score the text for educaitonal value.

The intended use of this dataset is to create augmented multilingual data for training a model. We recommend you run this data through for example, an LLM to extract useful information or reformat to stories, textbooks, etc.