greatnorth-us-federal-laws-text

# Great North US Federal Laws Text Corpus **Best quality structured + text corpus from the United States Code, CFR, and high-value US federal legal/policy documents.** This project turns the full (or as much as bulk sources allow) body of US federal law into training-ready data for sovereign AI. ## Sources (best quality) - United States Code (official XML via House OLRC USLM format) - individual titles for reliability (see fetch.py for @release-point zips). - Code of Federal Regulations (eCFR / govinfo bulk XML) - regulatory law. - High-value supplemental documents (major statutes, executive orders, congressional reports, etc.) via text extraction for the hybrid path (see greatnorth-gov-text-corpus). ## Output Training JSONL with clean text per section + rich metadata (citation, title, jurisdiction, date, source, source_url, retrieved_at, language, doc_type, optional chunk_index). Designed for pretraining legal/governance models and statistical analysis of US law. ## This Build (CFR focus) - 402,998 training records (section- and chunk-level). - ~1.49 GB JSONL. - ~737 MB raw official eCFR XML (titles 1-3,5-8,10,12-16,18-26,28-29,31-34,36,38,40-50 and others; full list in data/raw/us_laws_manifest.json). - US Code XML zips: correct download URLs in code (uscode.house.gov timed out in build env; rerun `fetch` when reachable to add tens/hundreds of k more USLM records from titles 1,2,3,5,10,15,18,21,28,31,42,44,49,50 etc.). All text is from official government structured XML (no OCR/PDF synthesis). Provenance preserved for every record. Public domain / U.S. Government works. ## Reproducibility Windows/PowerShell (recommended): ```powershell cd greatnorth-us-laws-corpus $env:PYTHONPATH="src" python -m pip install -e . --quiet python -m greatnorth_us_laws_corpus.cli fetch python -m greatnorth_us_laws_corpus.cli normalize # Upload (token via env only - never paste literal here) $env:HF_TOKEN=$env:HF_TOKEN python -m greatnorth_us_laws_corpus.cli upload --delete-after-upload ``` Raw bulk + processed training data both preserved until successful upload + delete (keeps disk clean for large legal text). ## Citation After upload the dataset will be at https://huggingface.co/datasets/GreatNorthCollective/greatnorth-us-federal-laws-text ## Graphify After code edits: graphify update . (from within project) Great North Collective. (2026). *Great North US Federal Laws Text Corpus* [Dataset]. Hugging Face. Contains information from the United States Government (uscode.house.gov, govinfo.gov), public domain / U.S. Government works. No endorsement implied. Use for training and analysis only; verify against official sources for legal purposes. ## Reproducibility & Upload (token via env var only) See commands in Reproducibility section above. Always reference token as `$env:HF_TOKEN=$env:HF_TOKEN ...` (never hard-code or paste the value in scripts/chat). After successful upload the CLI will delete local data/raw and data/processed.