Scientific-Summaries-KU-Fixes

# Scientific-Summaries — Knowledge-Unit Fixes A **difference index** of faulty knowledge units (rows) in [laion/Scientific-Summaries](https://huggingface.co/datasets/laion/Scientific-Summaries), plus everything needed to apply the corrections on load. The main dataset is not modified. Instead, this index lists only the faulty rows (2.5M of 22.1M) and what to do with them. Built against revision `232547b7b938a6588ce5d9d7faa96384860b7a16` of the main dataset (see `revision.txt`). All 7,907 shards were scanned; none failed. ## What was checked | Check | Result | |---|---| | Column names across all 7,907 shards | identical: the same 44 canonical keys everywhere | | JSON sub-keys of `claims` | **665 rows** deviate: an extra `caveats`/`caveat` key (444) or `limitations` key (3); misspelled keys `contradating_evidence`, `contrad_evidence`, `contradictions_evidence`, `contradicting evidence`; missing `contradicting_evidence` (3); empty list `[]` (214) | | JSON sub-keys of `source_authors`, `oa_authorships`, `oa_referenced_works` | fully consistent; no unparsable JSON in any column | | `summary_title` | **2,106,987** placeholders (`[Unknown]` etc.), 657 empty | | `summary_authors` | 107,408 placeholders, 16,072 "the authors are not provided"-style sentences, 1,333,094 empty | | Near-empty summaries (≥10 of 18 fields blank) | **7,221** rows, flagged `needs_regeneration` | | Column types | `arxiv/titled_*` has null-typed OpenAlex columns, and `arxiv/old_*` stores token counts as double. `unified_schema.json` fixes both | Faulty KUs per subset: | Subset | Faulty KUs | Share | Title | Authors | Needs regeneration | Claims | |---|---|---|---|---|---|---| | arxiv | 54,181 | 1.6% | 49,147 | 26,959 | 392 | 93 | | bethgelab | 3,301 | 1.0% | 3,103 | 749 | 39 | 7 | | pubmed | 460,552 | 26.6% | 366,189 | 335,535 | 3,033 | 39 | | web-papers | 1,989,353 | 11.9% | 1,689,205 | 1,093,331 | 3,757 | 526 | ## Files | File | Contents | |---|---| | `ku_fix_index.parquet` | one row per faulty KU (157 MB) | | `unified_schema.json` | one type per column, valid for every subset | | `json_key_stats.json` | every sub-key set seen in each JSON column, with counts, over all 22,089,530 rows | | `apply_fixes.py` | `load_fixed_shard(shard)`: streams a shard, casts it, applies the fixes | | `build_fix_index.py` | the scanner that produced the index (streams over HTTP; no full download needed) | | `DUPLICATE_REPORT.md` | **duplicate check over all 22.1M KUs**: 12,538 strong and 209,087 strong + probable duplicates to drop | | `duplicates.parquet` | every KU in a duplicate cluster, with the copy to keep (`keep`, `keep_strong`) | | `duplicate_stats.json` | all duplicate numbers per key, including the largest groups | | `build_duplicate_index.py`, `analyze_duplicates.py` | the duplicate scanner and the DuckDB clustering | ## Index columns | Column | Meaning | |---|---| | `file`, `row_index` | shard path in the main repo and 0-based row position: the join key | | `paper_id`, `subset` | for verification | | `title_fault` | `placeholder` / `empty` / `not_provided` / null | | `title_original`, `title_fixed`, `title_fixed_source` | fix taken from `oa_title` (OpenAlex) or `source_title` | | `authors_fault`, `authors_original`, `authors_fixed`, `authors_fixed_source` | fix taken from `oa_authorships` (OpenAlex) or `source_authors` (GROBID, noisier) | | `n_blank_summary_fields`, `blank_summary_fields`, `needs_regeneration` | empty summary fields | | `json_faults` | e.g. `claims:extra_keys=caveats`, `claims:missing_keys=contradicting_evidence` | | `claims_original`, `claims_fixed` | claims with sub-keys normalized to `description`, `supporting_evidence`, `contradicting_evidence`, `implications`. Misspelled keys are mapped back; off-schema keys (`caveats`, `limitations`) are dropped from `claims_fixed` but kept in `claims_original` | ## Limitations - **Most placeholder titles cannot be repaired from the data.** Only 483 of 2.1M `[Unknown]` titles have an `oa_title` or `source_title` to fall back on, and the public release has no full text for these rows. By default `apply_fixes.py` sets them to null, so they are no longer mistaken for real titles. About 5.5% are supplementary files (`…supp…`, `…_ESM…`) rather than papers. - Empty `summary_authors` are flagged but often correct, e.g. for documents with no listed authors. - Author names from `source_authors` come from GROBID parsing and can be wrong. `apply_fixes.py` only uses OpenAlex authors unless you opt in. ## Usage ```python from huggingface_hub import hf_hub_download import importlib.util path = hf_hub_download("laion/Scientific-Summaries-KU-Fixes", "apply_fixes.py", repo_type="dataset") spec = importlib.util.spec_from_file_location("apply_fixes", path) apply_fixes = importlib.util.module_from_spec(spec); spec.loader.exec_module(apply_fixes) index = apply_fixes.load_index() # downloads ku_fix_index.parquet once tbl = apply_fixes.load_fixed_shard("data/web-papers/part-00100.parquet", index=index) # or, with a local copy of the main repo: # tbl = apply_fixes.load_fixed_shard("data/pubmed/part-00010.parquet", index=index, local_root="./Scientific-Summaries") ``` Options: `title_sources`, `author_sources` (add `"source_authors"` to accept GROBID names), `null_placeholders=False` to keep placeholders. The fixed table gains two columns, `ku_needs_regeneration` and `ku_json_faults`. To only drop bad rows: ```python import pyarrow.parquet as pq, pyarrow.compute as pc idx = pq.read_table(hf_hub_download("laion/Scientific-Summaries-KU-Fixes", "ku_fix_index.parquet", repo_type="dataset")) regen = idx.filter(idx["needs_regeneration"]) # 7,221 KUs to re-summarize ```