pretraining-high-quality

# Dataset Card for Lapa High Quality Pretraining Dataset ## Dataset Description **Dataset Summary** This dataset is a high quality subset of pretraining corpus for Ukrainian language. It was filtered using 6 models, measuring different quality aspects of the data: - lapa-llm/alignment-score-model - Alignment - filtering for disinformation - lapa-llm/gec-score-model - Grammatical Correctness of the text - lapa-llm/fineweb-nemotron-edu-score - Educational Value of the text - lapa-llm/fineweb-mixtral-edu-score - Educational Value of the text - lapa-llm/manipulative-score-model - How manipulative is the text - lapa-llm/fasttext-quality-score - Text Coherence (how close is it to ELI5 from Reddit style of explanations) All models are available in this collection: https://huggingface.co/collections/lapa-llm/lapa-v012-pretraining We perform CDF binning and perform max ensembling, then select data only from the highest performing bucket, which is this dataset. Additional measure, `formula-score`, combines most of them into one metric: ```python def formula_score(item): item["formula-score"] = np.median([item["fineweb-nemotron-edu-score"], item["fineweb-mixtral-edu-score"], item["fasttext-quality-score"],]) * item["alignment-score"] * item["manipulative-score"] * item["gec-score"] return item ``` This provides a balanced measure of quality across all classifiers. **Languages** - Ukrainian (uk) ## Dataset Creation **Source Data** - Base dataset: Kobza, FinePDFs, FineWeb, UberText ## Considerations for Using the Data **Social Impact** Aims to strengthen the Ukrainian-language LLM ecosystem and improve accessibility of language technology for Ukrainian speakers. ## Citation **BibTeX** TBD ## License CC-BY-SA-4.0 --- *This dataset is part of the "Lapa" - Ukrainian LLM initiative to advance natural language processing for the Ukrainian language.*