Trim adaptor: adaptor tags were filtered.Drop low quality: low quality tags were filtered.Drop CopyNum = 1: tags of copy number = 1 were filtered.Distinct clean tags: different kinds of clean tags.
--- license: apache-2.0 task_categories: - text-generation language: - en tags: - data-juicer - pretraining size_categories: - 1M<n<10M --- # The Pile -- PubMed Central (refined by Data-Juicer) A re