ankitjh4-bharat-government-documents

# Bharat Guide: screened Indian public information documents This snapshot contains **64,964 distinct normalized text bodies** and 77,526 source records. Generated 2026-10-03T01:10:02.179230+00:00. ## Contents and provenance One Parquet row represents one normalized source body, with original extracted text, source title, URL, extraction quality, observed retrieval time, rule version, topic labels and all current-body source aliases. Duplicate URLs/editions are preserved as provenance, not counted as additional distinct documents. The canonical record is the lexicographically smallest eligible document identifier. Topic labels overlap. Selection requires an eligible assessment whose body checksum still matches the current document, with full_text or ocr_text extraction. This is heuristic screening, not manual factual validation. Older and newer screening versions coexist and are reported in manifest.json. PTI secondary news, directory-only entries, summary-only records, rejected text and OCR awaiting review are not qualifying rows. A reviewed alias can be retained inside an eligible row's provenance, explicitly labelled. The text exports contain no raw response bytes or account credentials. The combined CSV labels screened and review documents explicitly; failure records are separate. The separate SQLite backup retains original evidence and operational records; its manifest records its own snapshot date. ## Limitations This is not a complete census of Indian websites, laws, schemes or facilities. Government-adjacent institutional publications are labelled separately from government sources. Retrieval time is not publication time. Source validity, amendments, legal force, policy deadlines and numerical OCR accuracy require checking the original. Exact normalized-body deduplication does not establish semantic equivalence. ## Rights and use Rights remain with each original publisher. Government provenance does not establish that every source is public domain or has one common reuse licence. No blanket licence is granted over collected source text. Consult each source's terms and applicable rights before redistribution or other reuse. Publication does not change the rights retained by each original source. ## Releases manifest.json gives checksums and row counts. publication.json tracks confirmed publication content hashes and the additional-document threshold. Each update is committed atomically with its data and manifest; the next release is due after at least 10,000 newly qualifying distinct bodies, not 10,000 fetched URLs or chunks. ## Download layout and CSV fields `documents.csv.gz` is sorted by the full `url` value in ascending, case-sensitive order. `documents.csv.gz` is the main download: one deduplicated, nonempty extracted text body per row, combining screened and review content. The `status` column is `good` or `review`. Only `good` rows count toward the screened-document milestone. Review rows may include thin text, directory entries, news or incomplete extraction; keep them separate when evaluating quality. Duplicate source aliases are preserved in `provenance/sources.csv.gz` without repeating their text in the main CSV. The CSV contains: `content_hash`, `document_id`, `title`, `text`, `url`, `host`, `kind`, `trust`, `quality`, `state`, `category`, `fetched_at`, `metadata`, `sha256`, `status`, `assessment_version`, `assessment_status`, `assessment_reasons`, `topics`. `text` is the full stored extracted body, not a summary. `metadata` preserves the representative document's SQLite JSON metadata. `topics` is a JSON list, and labels may overlap; unclassified documents are explicitly labelled. Fetch time is not publication time. A row represents a document body, not every relational SQLite row. `topics/document_topics.csv.gz` maps topics to content hashes and good/review status; `topics/summary.json` provides counts. This index avoids copying full text into many folders. Topic labels are heuristic, not manual factual or legal validation. `fails/urls.csv.gz` contains failure, robots-blocked, retry and OCR-pending records with their exact statuses; not every entry is a permanent failure. `fails/empty_documents.csv.gz` lists records without usable text. Neither is counted as a distinct text document. Unattempted queued URLs and seed inventories are not exported. No crawler handoff package is included. CSV files use UTF-8, gzip compression and standard quoting for multiline text. Decompress to obtain an ordinary CSV, or read directly: ```python import pandas as pd df = pd.read_csv("documents.csv.gz") good = df[df["status"] == "good"] ``` Optional `data/train-*.parquet` files contain screened text in larger batches for the Hugging Face viewer. The CSV contains additional review content, so its row count is higher than the qualifying-document count. Source values are preserved; treat spreadsheet formulas in source text as untrusted, not executable instructions. ## SQLite reference [SQLITE_SCHEMA.md](SQLITE_SCHEMA.md) lists every table and field; `sqlite_schema.sql` contains the reference schema. `documents` stores text and metadata; `document_assessments` stores screening decisions; `document_topics` stores labels. Original PDFs and responses are in `raw_blobs.payload`, linked through `fetch_receipts.raw_sha256`, with `codec` describing zlib compression or raw bytes. `ocr_pages` stores page-level results. These binary files and operational tables are not embedded in the CSV. The private recovery files `backups/bharat.sqlite` and `backups/sqlite-manifest.json` retain their own snapshot date and checksum, which may precede the text release. The full SQLite backup includes crawl queues and source inventories even though this CSV release excludes them. Keep that backup private when publishing text-only data. ## Sources and subject coverage Sources include Indian central, state and local government portals; courts and legislative institutions; regulators and public-sector bodies; universities and research institutions; and other government-adjacent institutional sources. Domain endings alone do not establish ownership or reliability. Secondary news reporting, including PTI, is distinguished from official statements such as PIB releases. This release has text provenance from **16,122 distinct website hostnames**. Subdomains count separately. These are hosts represented in the released text, not a claim that every page of each website has been crawled. Failed-only, queued and seed-only hosts are excluded. | Website domain group | Distinct hosts | |---|---:| | .gov.in | 11,598 | | .ac.in | 1,791 | | Other .in | 2,283 | | Other domain endings | 450 | The current classifier defines **36 broad subject labels**, with the finer areas described below. The finer subject areas are descriptive; subtopic coverage is not counted independently. A website/document may appear under multiple topics, so topic totals must not be added together. Topic host counts include all released source aliases for labelled text, including `review` material. Labels are automated and may be inherited across identical-text aliases. `good` means passed extraction/source/subject heuristics, not manual fact-checking. Examples name institution types and subject areas, not a verified inventory of every named institution. | Topic | Website hosts | Good documents | Review documents | Source types and finer subject areas | |---|---:|---:|---:|---| | Law and judiciary | 2,009 | 8,908 | 11,438 | Courts, tribunals and legal-services bodies; Supreme Court, High Courts, judgments and legal aid | | Schemes and benefits | 3,471 | 10,755 | 4,590 | Central/state scheme portals; benefits, eligibility, applications and beneficiary guidance | | Policy and regulation | 5,891 | 8,385 | 8,247 | Ministries, departments and regulators; policies, regulations, guidelines and gazette notifications | | Government statements | 1,739 | 2,066 | 3,058 | Official information offices and PIB; press releases, cabinet decisions and public statements | | Universities and research | 3,410 | 9,844 | 9,169 | Universities, colleges and research institutions; admissions, curricula, accreditation and research | | Religion, heritage and endowments | 1,574 | 1,358 | 1,003 | Heritage and endowment bodies; religious institutions, pilgrimage, archaeology and endowments | | Mining and natural resources | 2,412 | 3,367 | 3,623 | Mining, geology and resource departments; minerals, coal, petroleum, water and forests | | Tourism | 1,491 | 3,567 | 3,417 | Tourism departments and public tourism bodies; destinations, visitor information and travel advisories | | Public finance and accounts | 2,775 | 2,951 | 4,578 | Finance departments, treasuries and audit bodies; expenditure, accounts, audits and appropriations | | Population and statistics | 1,045 | 972 | 1,008 | Census and statistics agencies; population, demographics, household surveys and vital statistics | | Banking and monetary policy | 625 | 1,527 | 1,146 | Banks and financial authorities; monetary policy, deposits, interest rates and financial stability | | Other public services | 3,742 | 8,463 | 7,448 | Citizen-service and sector departments; agriculture, public health, sanitation, drinking water and transport | | Constitution | 209 | 149 | 125 | Legislative and legal-information portals; constitutional text, provisions and constitutional materials | | Acts, amendments, rules and legal forms | 352 | 365 | 444 | Legislative departments and legal repositories; Acts, amendments, repeals, rules and statutory forms | | Hospitals and health facilities | 188 | 995 | 1,435 | Health departments and facility portals; hospitals, medical colleges, facility directories and infrastructure | | Income tax | 74 | 783 | 642 | Tax authorities; direct taxes, returns, deductions and taxpayer guidance | | Roads and public contracts | 238 | 945 | 1,202 | Roads departments and procurement portals; highways, tenders, contracts and awards | | Public infrastructure | 2,884 | 3,566 | 3,538 | Public works and infrastructure agencies; railways, urban development and public projects | | Digital infrastructure | 395 | 1,401 | 744 | Digital-government agencies; e-governance, broadband, BharatNet and digital public infrastructure | | ONDC | 4 | 13 | 20 | Digital-commerce network information; ONDC and open-network participation | | UPI and payment systems | 18 | 5 | 13 | Payment-system institutions; UPI, NPCI and payment-system documentation | | Prisons and correctional administration | 290 | 792 | 1,599 | Prison and corrections departments; administration, jail manuals and aggregate reports | | Juvenile justice | 35 | 147 | 97 | Child-welfare and justice bodies; legislation, observation homes and institutional reports | | Reproductive health and rights | 40 | 49 | 25 | Health and rights-related public bodies; maternal health, family planning and pregnancy-termination frameworks | | Minority rights | 73 | 315 | 936 | Minority-affairs departments and commissions; protections, programmes and public reports | | Culture and arts | 645 | 2,676 | 2,963 | Culture departments and public cultural institutions; museums, arts and cultural programmes | | Fire services and stations | 227 | 374 | 321 | Fire and emergency-service departments; stations, fire services and fire-safety guidance | | Police services and stations | 604 | 1,473 | 1,241 | Police departments; stations, public services and departmental information | | Credit and lending | 77 | 167 | 175 | Banks and credit-programme bodies; loans, guarantees, lending schemes and microfinance | | Government grants and sponsorships | 301 | 815 | 612 | Grant-making and education departments; grants, scholarships and sponsorships | | Government hospitality and hotels | 59 | 647 | 720 | Public tourism corporations and hospitality bodies; government hotels and state guest houses | | Official contact directories | 513 | 416 | 232 | Official departmental directories; office contacts, telephone directories and who-is-who pages | | International policy and statements | 128 | 320 | 403 | External-affairs and diplomatic institutions; bilateral relations, foreign policy and joint statements | | Parliament and briefings | 585 | 658 | 1,420 | Parliamentary institutions; Lok Sabha, Rajya Sabha, briefings and parliamentary materials | | Prime ministers: history and tenures | 1 | 15 | 6 | Official historical and institutional collections; former prime ministers and their tenures | | Budgets | 875 | 717 | 1,611 | Finance and budget portals; budgets, demands for grants and appropriation documents | | News reporting | 1 | 0 | 11 | Secondary news agencies, including PTI; news reporting kept distinct from official government statements | | Unclassified | 13,101 | 0 | 253,560 | Extracted material without a subject label; retained for review, not assigned an invented topic | Agriculture currently sits within **Other public services**; it does not yet have a separately measured website count. Zero-count topics indicate tracked areas without labelled text in this snapshot, not completed coverage. Hospital records are not a reconciled national inventory; legal materials may include superseded versions. For exact source URLs and titles, see [`provenance/sources.csv.gz`](provenance/sources.csv.gz). Document-level topics are in `documents.csv.gz`; the topic-to-document index and counts are in [`topics/`](topics/). These files describe released material and do not include queued or seed-only website lists.