TeleChat-PTD

"TeleChat-PTD is a comprehensive large-scale Chinese data set extracted from the TeleChat pre-training corpus of the Telecom Xingchen large model. The data mainly comes from web pages, books, official media, etc. We used the rule + model method to conduct relevant Filter and perform similarity deduplication on the data to extract high-quality data as much as possible. The TeleChat-PTD data set exposes approximately 270 million pieces of data. The data consists of pure Chinese text, with an original size of approximately 1TB and a compressed size of 480G, with a total of 189 files. Other redundant information has been removed from the data set. "