TucanoBR/GigaVerbo-Text-Filter

GigaVerbo Text-Filter是一个包含110,000个随机选择的样本的数据集，这些样本来自9个子集的GigaVerbo（即那些不是合成的子集）。这个数据集用于训练在论文《Tucano: Advancing Neural Text Generation for Portuguese》中描述的文本质量过滤器。为了创建文本嵌入，我们使用了sentence-transformers/LaBSE。所有的分数都是由GPT-4o生成的。

GigaVerbo Text-Filter is a dataset with 110,000 randomly selected samples from 9 subsets of GigaVerbo (i.e., specifically those that were not synthetic). This dataset was used to train the text-quality filters described in "[Tucano: Advancing Neural Text Generation for Portuguese](https://arxiv.org/abs/2411.07854)". To create the text embeddings, we used [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE). All scores were generated by GPT-4o.