C10X/finepdfs-edu-hq

# FinePDFs-Edu (English) — Filtered High-Signal Subset This dataset is a **filtered, English-only** subset of **HuggingFaceFW/finepdfs-edu**, created to retain **high-signal educational passages** while reducing common PDF-extraction noise (covers/TOCs, fragmented headers/footers, OCR artifacts, mixed-language pages, and very short low-context snippets). It is intended for training and research workflows that benefit from **longer, coherent educational text** extracted from PDFs. --- ## At a glance | Metric | Value | |--------|-------| | **Source Dataset** | HuggingFaceFW/finepdfs-edu | | **Language** | English (eng_Latn) | | **Total Processed** | 23,023,372 rows | | **Filtered (Kept)** | 5,512,514 rows (23.94%) | | **Rejected** | 17,510,858 rows (76.06%) | | **Total Tokens** | 28,252,158,603 | | **Average Tokens** | 5,125 | | **Output Shards** | 55 | | **Output Size** | 38.37 GB | --- ## Why this dataset exists FinePDFs-Edu is large and valuable, but PDF-to-text extraction naturally includes many borderline samples. In practice, many projects want a more “ready-to-train” slice with: - **Language consistency** (English-only with high confidence) - **Sufficient length** (avoid short fragments that behave like noise) - **Minimum educational usefulness** (filter out low-signal content) This release provides a simple, transparent filtering recipe plus reproducible logs and summary statistics. --- ## Filtering strategy Four filters were applied. Each targets a common failure mode in PDF-derived text. ### 1) English-only documents - `full_doc_lid == "eng_Latn"` This enforces a single dominant language at the document level. ### 2) High-confidence language detection - `page_average_lid_score >= 0.9` Using a page-averaged confidence score helps reduce multilingual leakage and typical OCR/segmentation issues. > Note: `full_doc_lid_score` is preserved in the schema for analysis, but this release uses **page-level confidence** as the primary LID quality signal. ### 3) Minimum content length - `token_count >= 512` In PDF corpora, very short samples are often headers/menus/fragmented text. A 512-token minimum strongly reduces these low-context snippets. ### 4) Minimum educational quality - `fw_edu_scores >= 2.0` This threshold aims to keep content that is at least moderately educational, while still preserving breadth. --- ## Data quality snapshots These binned summaries make it easy to sanity-check the retained distribution. ### Language ID confidence (page_average_lid_score) | Range | Count | Percentage | Visual | |-------|--------|------------|--------| | 0.90-0.92 | 246,335 | 4.5% | ██ | | 0.92-0.94 | 312,234 | 5.7% | ██ | | 0.94-0.96 | 404,114 | 7.3% | ███ | | 0.96-0.98 | 581,289 | 10.5% | █████ | | **0.98-1.00** | **3,968,537** | **72.0%** | ███████████████████████████████████ | **Interpretation:** The distribution is heavily concentrated in **0.98–1.00**, indicating very high English confidence after filtering. --- ## Token distribution | Metric | Value | |--------|-------| | Total Tokens | 28,252,158,603 | | Average | 5,125 | | Minimum | 512 | | Maximum | 12,113,077 | **Interpretation:** Average length suggests medium-to-long passages (often multi-page sections). Some documents are extremely long; many pipelines will prefer to chunk or span-sample at load time. --- ## Educational score notes (fw_edu_scores) `fw_edu_scores` is a heuristic educational-value score. In practice, **lower values often correlate** with noisier extraction or weaker instructional structure, while **higher values often correlate** with more structured, tutorial- or textbook-like writing. Important nuance: the rubric is oriented toward **school-level educational value**, so highly technical/advanced material may not always receive the highest score even if it is high quality. --- ## Intended use cases - **Continued pretraining / domain adaptation** toward educational writing styles - **Fine-tuning** on longer instructional passages (summarization, explanation, QA, classification) - **Research** on PDF extraction, educational text quality, and filtering strategies --- ## Reproducibility (kept logs) Filtering configuration (as used to produce this release): - Dataset: `HuggingFaceFW/finepdfs-edu` - Subset: `eng_Latn` - Filters: - `full_doc_lid = eng_Latn` - `page_average_lid_score >= 0.9` - `token_count >= 512` - `fw_edu_scores >= 2.0` - Output: - Sharded Parquet (~100,000 rows per shard) - Compression: `zstd` --- ## Performance This run was executed on a CPU-only setup: - **CPU:** 4 cores / 8 threads | Stage | Duration | |-------|----------| | Dataset Load | 2.29s | | Processing | 21,558s (~6 hours) | | **Total** | **21,560s** | | Processing Speed | ~1,068 rows/second | --- ## Schema (key columns) The dataset follows the original FinePDFs-Edu schema. The most commonly used columns are: | Column | Type | Description | |--------|------|-------------| | `text` | string | Document text | | `id` | string | Unique identifier | | `token_count` | int64 | Token count | | `page_average_lid` | string | Page-averaged language ID | | `page_average_lid_score` | float64 | Page-averaged LID confidence | | `full_doc_lid` | string | Document-level language ID | | `full_doc_lid_score` | float64 | Document-level LID confidence | | `fw_edu_scores` | float64 | Educational score | | `minhash_cluster_size` | int64 | Cluster size proxy (dedupe signal) | | `duplicate_count` | int64 | Duplicate count | For the complete schema, see the dataset viewer / parquet schema. --- ## Storage layout ``` optimized/ ├── README.md ├── dataset_info.json ├── shard-00000.parquet ├── shard-00001.parquet ├── ... └── shard-00054.parquet ``` --- ## Usage examples ### Fast analytics with DuckDB ```python import duckdb con = duckdb.connect() stats = con.execute(""" SELECT COUNT(*) AS rows, AVG(token_count) AS avg_tokens, AVG(page_average_lid_score) AS avg_lid, AVG(fw_edu_scores) AS avg_edu FROM 'optimized/*.parquet' """).fetchall() print(stats) # Optional: create a cleaner slice con.execute(""" COPY ( SELECT id, text FROM 'optimized/*.parquet' WHERE fw_edu_scores >= 3.0 AND token_count >= 1024 ) TO 'train_slice.parquet' (FORMAT PARQUET); """) ``` --- ## Citation If you use this dataset, please cite the original FinePDFs dataset: ```bibtex @misc{kydlicek2025finepdfs, title={FinePDFs}, author={Kydlíček, Hynek and Penedo, Guilherme and von Werra, Leandro}, year={2025}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu}} } ``` --- ## License This filtered dataset inherits the **ODC-By v1.0** license from the original FinePDFs-Edu dataset. --- ## Changelog - **2026-01-18**: Initial filtered release (English-only, high-confidence LID, min length 512 tokens, min edu score 2.0) --- **Generated by:** FinePDFs-Edu Dataset Streamer **Generated at:** 2026-01-18