davanstrien/finepdfs-eng-stats
收藏Hugging Face2025-12-11 更新2025-12-20 收录
下载链接：
https://hf-mirror.com/datasets/davanstrien/finepdfs-eng-stats
下载链接
链接失效反馈官方服务：
资源简介：
该数据集是对HuggingFaceFW/finepdfs-edu数据集的聚合统计结果，使用Polars流式处理技术计算得出。统计内容包括全局摘要（如总文档数、总令牌数、平均令牌数/文档等）、按语言、提取器和转储的详细统计。处理了23,023,372个文档，总用时2608.95秒。数据集展示了Polars流式聚合与HuggingFace Hub集成的能力。
Aggregate statistics computed using Polars streaming on the [HuggingFaceFW/finepdfs-edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu) dataset. The statistics include global summaries (total documents, total tokens, average tokens per document, etc.), detailed statistics by language, extractor, and dump. Processed 23,023,372 documents in 2608.95 seconds. Demonstrates Polars streaming aggregation with HuggingFace Hub integration.
提供机构：
davanstrien
