biglam/hmd_newspapers

---
dataset_info:
features:
- name: source
dtype: string
- name: title
dtype: string
- name: location
dtype: string
- name: date
dtype: timestamp[s]
- name: item_type
dtype: string
- name: word_count
dtype: int32
- name: ocr_quality_mean
dtype: float64
- name: ocr_quality_sd
dtype: float64
- name: text
dtype: string
splits:
- name: train
num_bytes: 14304741164
num_examples: 3065408
download_size: 9682476047
dataset_size: 14304741164
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
license: cc0-1.0
task_categories:
- text-generation
language:
- en
tags:
- newspapers
pretty_name: Heritage Made Digital Newspapers
size_categories:
- 1M<n<10M
---
# Dataset Card for Heritage Made Digital Newspapers

## Table of Contents
- [Dataset Card for Heritage Made Digital Newspapers](#dataset-card-for-heritage-made-digital-newspapers)
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** https://bl.iro.bl.uk/?locale=en
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

This dataset contains text extracted at the article level from historic digitised newspapers from the [Heritage Made Digital](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en) newspaper digitisation program at the [British Library](https://www.bl.uk/). The newspapers in the dataset were published between 1800 and 1896. This dataset contains ~2.5 billion tokens and 3,065,408 articles.

The dataset contains text generated from Optical Character Recognition software on digitised newspaper pages. This dataset includes the plain text from the OCR alongside some minimal metadata associated with the newspaper from which the text is derived and OCR confidence score information generated from the OCR software.

### Supported Tasks and Leaderboards

This dataset can be used for:
- historical research and digital humanities research
- training language models
- training historic language models

Whilst this dataset can be used for all of these tasks, it is important to understand that the dataset was not constructed in a representative way so it contains biases in terms of the newspapers and articles that are included (more on this below).

### Languages

The text in this dataset is in English that has been recognised by the OCR software. The OCR software used is generic commercial OCR software that has not been trained on historic newspapers. There are therefore many errors in the text. Some of the OCR in this text will be of such poor quality that is is incomprehensible to a human reader.

## Dataset Structure

### Data Instances

Each row in the dataset is an article from a newspaper as recognised by an OLR (Optical Layout Recognition) step in the digitisation process.

### Data Splits

There is one split in this dataset, the training split.

## Dataset Creation

### Curation Rationale

This dataset consists of public-domain newspapers published in the UK during the 19th Century. The majority of newspapers digitised in the UK are not freely available (even if they are out of copyright). The newspapers in this dataset were digitised specifically to be freely available but also to meet preservation goals for newspapers in poor condition. As a result, the newspapers chosen for digitisation are biased toward poor quality physical newspapers. This may in turn result in worse OCR.

### Source Data

The source data for this dataset is the digitised newspapers from the [Heritage Made Digital](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en) newspaper digitisation program. The newspapers in the dataset were published between 1800 and 1870.

### Dataset Curators

The original digitisation was carried out by the British Library. The dataset was created by the British Library in partnership with Findmypast.

This dataset was created by [@davanstrien](https://huggingface.co/davanstrien).

### Licensing Information

The newspapers in this dataset are in the public domain. The dataset is licensed under a [Creative Commons Zero v1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/) license.

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@github-username](https://github.com/<github-username>) for adding this dataset.

数据集信息：
特征：
- 名称：来源
数据类型：字符串
- 名称：标题
数据类型：字符串
- 名称：出版地点
数据类型：字符串
- 名称：出版日期
数据类型：时间戳（单位：秒，timestamp[s]）
- 名称：条目类型
数据类型：字符串
- 名称：词数
数据类型：32位整数（int32）
- 名称：OCR质量均值（ocr_quality_mean）
数据类型：64位浮点数（float64）
- 名称：OCR质量标准差（ocr_quality_sd）
数据类型：64位浮点数（float64）
- 名称：文本
数据类型：字符串
数据划分：
- 名称：训练集（train）
字节数：14304741164
样本数：3065408
下载大小：9682476047
数据集总大小：14304741164
配置：
- 配置名称：默认（default）
数据文件：
- 划分：训练集
路径：data/train-*
许可协议：CC0-1.0
任务类别：
- 文本生成（text-generation）
语言：
- 英语（en）
标签：
- 报纸（newspapers）
易读名称：遗产数字化报纸（Heritage Made Digital Newspapers）
样本规模区间：1M<n<10M

# 遗产数字化报纸（Heritage Made Digital Newspapers）数据集卡片

## 目录
- [遗产数字化报纸数据集卡片](#遗产数字化报纸数据集卡片)
- [目录](#目录)
- [数据集说明](#数据集说明)
- [数据集概述](#数据集概述)
- [支持任务与基准测试榜](#支持任务与基准测试榜)
- [语言说明](#语言说明)
- [数据集结构](#数据集结构)
- [数据实例](#数据实例)
- [数据划分](#数据划分)
- [数据集构建](#数据集构建)
- [筛选依据](#筛选依据)
- [源数据](#源数据)
- [数据集策展人](#数据集策展人)
- [许可信息](#许可信息)
- [引用信息](#引用信息)
- [贡献致谢](#贡献致谢)

## 数据集说明

- **主页：**
- **仓库地址：** https://bl.iro.bl.uk/?locale=en
- **相关论文：**
- **基准测试榜：**
- **联系人：**

### 数据集概述
本数据集提取自大英图书馆（British Library）的[遗产数字化报纸项目（Heritage Made Digital）](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en)的历史数字化报纸的文章级文本。数据集内收录的报纸出版时间跨度为1800年至1896年，总计包含约25亿个标记（Token）与3065408篇文章。

本数据集的文本来自对数字化报纸页面的光学字符识别（Optical Character Recognition, OCR）结果，包含OCR生成的纯文本、来源报纸的极简元数据，以及OCR软件生成的OCR置信度评分信息。

### 支持任务与基准测试榜
本数据集可应用于以下场景：
- 历史研究与数字人文研究
- 语言模型训练
- 历史语言模型训练

尽管本数据集可适用于上述所有任务，但需注意：该数据集并非按代表性原则构建，因此收录的报纸与文章存在偏差（详细说明见下文）。

### 语言说明
本数据集的文本为OCR软件识别的英语。所用OCR软件为通用商业OCR软件，未针对历史报纸语料进行定制训练，因此文本中存在大量识别错误。部分OCR结果质量极差，人类读者无法理解其内容。

## 数据集结构

### 数据实例
数据集中的每一行均为数字化流程中通过光学布局识别（Optical Layout Recognition, OLR）步骤识别出的单篇报纸文章。

### 数据划分
本数据集仅包含一个数据划分，即训练集。

## 数据集构建

### 筛选依据
本数据集收录19世纪英国出版的公有领域报纸。英国境内多数完成数字化的报纸即使已过版权保护期，也并非免费开放。本数据集的报纸经专门数字化处理，既实现免费公开，同时也满足对物理状态破损严重的报纸的保存需求。因此，入选数字化的报纸偏向质量较差的藏品，这可能进一步导致OCR识别质量下降。

### 源数据
本数据集的源数据来自[遗产数字化报纸项目（Heritage Made Digital）](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en)的数字化报纸馆藏。数据集内收录的报纸出版时间跨度为1800年至1870年。

### 数据集策展人
原始数字化工作由大英图书馆完成。本数据集由大英图书馆与Findmypast合作创建。

本数据集由[@davanstrien](https://huggingface.co/davanstrien)整理上传。

### 许可信息
数据集内收录的报纸均属于公有领域。本数据集采用[知识共享零许可v1.0（Creative Commons Zero v1.0 Universal）](https://creativecommons.org/publicdomain/zero/1.0/)协议授权。

### 引用信息
[需补充更多信息]

### 贡献致谢
感谢[@github-username](https://github.com/<github-username>)添加本数据集。

原始信息汇总

数据集概述

数据集名称

• 名称: Heritage Made Digital Newspapers

数据集特征

• 特征列表:
  • source: 字符串类型
  • title: 字符串类型
  • location: 字符串类型
  • date: 时间戳类型
  • item_type: 字符串类型
  • word_count: 整数类型
  • ocr_quality_mean: 浮点数类型
  • ocr_quality_sd: 浮点数类型
  • text: 字符串类型

数据集结构

• 数据分割:
  • train: 训练集，包含3,065,408个样本，总大小为14,304,741,164字节

数据集大小

• 下载大小: 9,682,476,047字节
• 数据集总大小: 14,304,741,164字节

许可证

• 许可证: Creative Commons Zero v1.0 Universal

任务类别

• 任务类别: 文本生成

语言

• 语言: 英语

标签

• 标签: 报纸

数据集创建

• 创建者: 英国图书馆与Findmypast合作创建
• 原始数字化: 英国图书馆执行
• 数据集创建者: @davanstrien

数据集用途

• 用途:
  • 历史研究与数字人文研究
  • 训练语言模型
  • 训练历史语言模型

数据集描述

• 描述: 该数据集包含从19世纪的历史数字化报纸中提取的文章级文本，主要用于历史研究和语言模型训练。数据集包含约25亿个令牌和3,065,408篇文章，文本由光学字符识别软件生成，伴随有来自报纸的元数据和OCR软件生成的置信度分数信息。