uk_hansard

# UK Hansard

## Description
Hansard represents the official record of parliamentary proceedings across the United Kingdom’s legislative bodies.
This dataset incorporates records from multiple sources, including debates and written answers from the UK Commons and Lords, devolved legislatures (Scottish Parliament, Senedd in both English and Welsh, Northern Ireland Assembly), London Mayor’s Questions, and ministerial statements.
Data was sourced from [ParlParse](https://parser.theyworkforyou.com/), covering Commons debates from 1918 forward and Lords proceedings from the 1999 reform.
Each document was processed to preserve complete parliamentary sessions as cohesive units, maintaining the natural flow of debate.
All content is published under the [Open Parliament License](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/).

## Dataset Statistics
| Documents | UTF-8 GB |
|-----------|----------|
| 51,552 | 10 |

## License Issues
While we aim to produce datasets with completely accurate licensing information, license laundering and inaccurate metadata can cause us to erroneously assign the incorrect license to some documents (for further discussion of this limitation, please see [our paper](https://huggingface.co/papers/2506.05209)). If you believe you have found an instance of incorrect licensing in this dataset, please [start a discussion](https://github.com/r-three/common-pile/discussions/new) on this repository.

## Other Versions
This is the "raw" version of the UK Hansard dataset.
If you are looking for the filtered version used to train [Comma v0.1](https://huggingface.co/common-pile/comma-v0.1), you can find it [here](https://huggingface.co/datasets/common-pile/uk_hansard_filtered).

## Citation
If you use this dataset, please cite:
```bibtex
@article{kandpal2025common,
title={{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
author={Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben and Elie Bakouch and John David and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
journal={arXiv preprint},
year={2025}
}
```
<!--
# UK Hansard Dataset

## Overview

This dataset contains the official records of parliamentary proceedings across the United Kingdom's legislative bodies, known as Hansard. It represents a comprehensive collection of parliamentary discourse spanning multiple decades and legislative bodies
.
## Specifications

- **Document Count**: 51,552 documents
- **Total Size**: ~10GB
- **Tokens**: ~2.5B
- **Time Range**: Records spanning from 1918 (Commons debates) and 1999 (Lords proceedings) to present
-
## Description

The UK Hansard Dataset incorporates records from multiple sources:

- **UK House of Commons debates** (from 1918 forward)
- **UK House of Lords proceedings** (from the 1999 reform forward)
- **Devolved legislatures:**
- Scottish Parliament
- Senedd (Welsh Parliament) in both English and Welsh
- Northern Ireland Assembly
- **London Mayor's Questions**
- **Ministerial statements**

All data was sourced from ParlParse, a parliamentary data parsing project that makes these records accessible in structured formats.

## Data Processing

Each document in this dataset has been processed to preserve complete parliamentary sessions as cohesive units, preserving the original question-answer format of parliamentary proceedings.

## Dataset Structure

Each document in the dataset is structured as follows:

- **text**: The full text content of the parliamentary proceeding
- **source**: Source identifier of the document
- **id**: Unique identifier for the document
- **added**: Timestamp when the document was added to the dataset
- **metadata**: A dictionary containing additional information such as:
- Year of publication
- Language (primarily English, with Welsh content where applicable)
- License information
## Acknowledgments

This work builds on the [ParlParse project](https://data.mysociety.org/datasets/uk-hansard/), which has made UK parliamentary data more accessible to researchers and the public.

## License

[Open Parliament License](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/)
-->

# 英国议会汉萨德记录（UK Hansard）

## 数据集描述
汉萨德（Hansard）是英国各级立法机构议会议事的官方记录。本数据集整合了多源记录，涵盖英国下议院、上议院的辩论与书面答复，权力下放立法机构（苏格兰议会、同时提供英文与威尔士语内容的威尔士议会（Senedd）、北爱尔兰议会）、伦敦市长质询环节以及大臣声明的相关内容。
数据集源自[ParlParse](https://parser.theyworkforyou.com/)，覆盖1918年起的下议院辩论记录，以及1999年议会改革后的上议院议事记录。所有文档均经过处理，以完整的议会会议为连贯单元进行保存，保留辩论的自然行文逻辑。所有内容均遵循[开放议会许可（Open Parliament License）](https://www.parliament.uk/site-information/copyright-parliament/open-parliament-licence/)发布。

## 数据集统计
| 文档数量 | UTF-8编码大小（GB） |
|-----------|----------|
| 51,552 | 10 |

## 许可相关问题
尽管我们致力于提供完全准确的许可信息，但由于许可洗白（license laundering）与元数据（metadata）不准确的问题，我们可能会错误地为部分文档分配了错误的许可协议。如需进一步了解该局限性的相关讨论，请参阅[我们的论文](https://huggingface.co/papers/2506.05209)。若您发现本数据集存在许可信息错误的情况，请前往本仓库[发起讨论](https://github.com/r-three/common-pile/discussions/new)。

## 其他版本
本版本为英国议会汉萨德数据集的“原始版”。若您需获取用于训练[Comma v0.1](https://huggingface.co/common-pile/comma-v0.1)的过滤版数据集，可前往[此处](https://huggingface.co/datasets/common-pile/uk_hansard_filtered)获取。

## 引用格式
若您使用本数据集，请引用以下文献：
bibtex
@article{kandpal2025common,
title={{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
author={Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben and Elie Bakouch and John David and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
journal={arXiv preprint},
year={2025}
}