m-a-p/FineFineWeb-bert-seeddata
收藏Hugging Face2024-12-19 更新2025-02-15 收录
下载链接：
https://hf-mirror.com/datasets/m-a-p/FineFineWeb-bert-seeddata
下载链接
链接失效反馈官方服务：
资源简介：
FineFineWeb是一个关于细粒度领域网络语料库的全面研究，旨在为各种任务提供全面的领域特定数据。该数据集通过精确去重和MinHash技术去重，使用GPT-4对根URL进行计数和标记，生成领域兴趣和不兴趣的粗粒度URL作为种子数据源。通过多轮粗召回和细召回迭代，结合FastText和BERT模型，最终构建出细粒度的领域数据集。
FineFineWeb is a comprehensive study on fine-grained domain web corpus, aiming to provide domain-specific data for a variety of tasks. The dataset is deduplicated using exact deduplication and MinHash techniques, counted and labeled root URLs using GPT-4, generating coarse-grained URLs of domain interest and non-interest as seed data sources. Through multiple rounds of coarse recall and fine recall iterations, combining FastText and BERT models, a fine-grained domain dataset is constructed.
提供机构：
m-a-p
