HiTab-StatCan-NSF

# HiTab-StatCan-NSF A commercially-permissive subset of [Microsoft HiTab](https://github.com/microsoft/HiTab) (ACL 2022) containing only the Statistics Canada and NSF table sources. The Wikipedia/ToTTo subset (CC-BY-SA-4.0, share-alike) is excluded. ## License - **NSF tables** (~14% of rows): sourced from US National Science Foundation reports. US federal government works are public domain under 17 U.S.C. § 105. - **StatCan tables** (~86% of rows): sourced from Statistics Canada reports under the [Statistics Canada Open Government Licence](https://www.statcan.gc.ca/en/reference/licence), which permits commercial use with attribution. Cite as: "Adapted from Statistics Canada. Reproduced and distributed on an 'as is' basis with the permission of Statistics Canada." The original HiTab dataset is released under C-UDA-1.0. That agreement's Section 1.3 preserves your rights under any other applicable licence, so the per-source terms above govern the StatCan and NSF subsets. ## Source Built from [microsoft/HiTab](https://github.com/microsoft/HiTab) by filtering `table_source` to `statcan` and `nsf`. ## Content Hierarchical table question answering and table-to-text generation covering Canadian statistical reports (demographics, immigration, economics, health) and US NSF science and engineering statistics. Tables have multi-level row and column headers encoded as HTML with `rowspan`/`colspan` and as raw JSON. ## Splits | Split | Rows | StatCan | NSF | |---|---|---|---| | train | 6,041 | 5,486 | 555 | | validation | 1,378 | 1,268 | 110 | | test | 1,286 | 1,189 | 97 | | **total** | **8,705** | **7,943** | **762** | ## Schema | Field | Type | Description | |---|---|---| | `id` | string | Original HiTab MD5 row identifier | | `table_id` | string | Table file key (matches HiTab `tables/hmt/`) | | `table_source` | string | `statcan` or `nsf` | | `question` | string | Natural language question | | `answer` | list[string] | Answer values (numbers and strings both as strings) | | `aggregation` | list[string] | Aggregation operation(s) required | | `sub_sentence` | string | Reference table-to-text sentence | | `sentence_id` | string | HiTab sentence group identifier | | `sub_sentence_id` | string | HiTab sub-sentence identifier | | `linked_cells` | string (JSON) | Cell-to-entity/quantity linkage map | | `answer_formulas` | list[string] | Spreadsheet-style formulas for answers | | `reference_cells_map` | string (JSON) | Formula cell reference map | | `table_title` | string | Full table title | | `table_html` | string | Table rendered as HTML with hierarchical rowspan/colspan headers | | `table_json` | string | Full table as JSON (title, top_root, left_root, data trees) | Use `table_source` to filter to a single provenance if your licence constraints require it. ## Usage ```python from datasets import load_dataset # All clean rows ds = load_dataset("rootsautomation/HiTab-StatCan-NSF") # Public-domain only (NSF) nsf = ds.filter(lambda x: x["table_source"] == "nsf") # StatCan only statcan = ds.filter(lambda x: x["table_source"] == "statcan") ``` ## Citation If you use this dataset please cite the original HiTab paper: ```bibtex @inproceedings{cheng-etal-2022-hitab, title = {{HiTab}: A Hierarchical Table Dataset for Question Answering and Natural Language Generation}, author = {Cheng, Zhoujun and Dong, Haoyu and Wang, Zhiruo and Jia, Ran and Guo, Jiaqi and Gao, Yan and Han, Shi and Lou, Jian-Guang and Zhang, Dongmei}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2022}, pages = {867--880}, } ```