neuclir/hc4
收藏资源简介：
--- annotations_creators: - no-annotation language: - fa - ru - zh language_creators: - found license: - odc-by multilinguality: - multilingual pretty_name: HC4 size_categories: - 1M<n<10M source_datasets: - extended|c4 tags: [] task_categories: - text-retrieval task_ids: - document-retrieval --- # Dataset Card for HC4 ## Dataset Description - **Repository:** https://github.com/hltcoe/HC4 - **Paper:** https://arxiv.org/abs/2201.09992 ### Dataset Summary HC4 is a suite of test collections for ad hoc Cross-Language Information Retrieval (CLIR), with Common Crawl News documents in Chinese, Persian, and Russian. The documents are Web pages from Common Crawl in Chinese, Persian, and Russian. ### Languages - Chinese - Persian - Russian ## Dataset Structure ### Data Instances | Split | Documents | |-----------------|----------:| | `fas` (Persian) | 486K | | `rus` (Russian) | 4.7M | | `zho` (Chinese) | 646K | ### Data Fields - `id`: unique identifier for this document - `cc_file`: source file from connon crawl - `time`: extracted date/time from article - `title`: title extracted from article - `text`: extracted article body - `url`: source URL ## Dataset Usage Using 🤗 Datasets: ```python from datasets import load_dataset dataset = load_dataset('neuclir/hc4') dataset['fas'] # Persian documents dataset['rus'] # Russian documents dataset['zho'] # Chinese documents ``` ## Citation Information ``` @article{Lawrie2022HC4, author = {Dawn Lawrie and James Mayfield and Douglas W. Oard and Eugene Yang}, title = {HC4: A New Suite of Test Collections for Ad Hoc CLIR}, booktitle = {{Advances in Information Retrieval. 44th European Conference on IR Research (ECIR 2022)}, year = {2022}, month = apr, publisher = {Springer}, series = {Lecture Notes in Computer Science}, site = {Stavanger, Norway}, url = {https://arxiv.org/abs/2201.09992} } ```
annotations_creators: - 无注释 language: - 波斯语（fa） - 俄语（ru） - 中文（zh） language_creators: - 公开获取（found） license: - 开放数据Commons署名许可（odc-by） multilinguality: - 多语言 pretty_name: HC4 size_categories: - 100万<数据规模<1000万 source_datasets: - 扩展|c4 tags: [] task_categories: - 文本检索（text-retrieval） task_ids: - 文档检索（document-retrieval） # HC4 数据集卡片 ## 数据集描述 - **代码仓库**: https://github.com/hltcoe/HC4 - **相关论文**: https://arxiv.org/abs/2201.09992 ### 数据集概览 HC4 是一套用于即席跨语言信息检索（Cross-Language Information Retrieval, CLIR）的测试集套件，包含来自公共爬虫（Common Crawl）的中文、波斯语及俄语新闻网页文档。所有文档均为取自公共爬虫的中文、波斯语与俄语网页。 ### 支持语言 - 中文 - 波斯语 - 俄语 ## 数据集结构 ### 数据实例 | 数据拆分 | 文档数量 | |----------------|---------:| | `fas`（波斯语） | 48.6万 | | `rus`（俄语） | 470万 | | `zho`（中文） | 64.6万 | ### 数据字段 - `id`: 文档的唯一标识符 - `cc_file`: 来自公共爬虫的源文件 - `time`: 从文章中提取的发布日期/时间 - `title`: 从文章中提取的标题 - `text`: 提取的文章正文 - `url`: 源网页链接 ## 数据集使用方法 使用 Hugging Face 🤗 数据集库： python from datasets import load_dataset dataset = load_dataset('neuclir/hc4') dataset['fas'] # 波斯语文档 dataset['rus'] # 俄语文档 dataset['zho'] # 中文文档 ## 引用信息 @article{Lawrie2022HC4, author = {Dawn Lawrie and James Mayfield and Douglas W. Oard and Eugene Yang}, title = {HC4: A New Suite of Test Collections for Ad Hoc CLIR}, booktitle = {{Advances in Information Retrieval. 44th European Conference on IR Research (ECIR 2022)}, year = {2022}, month = apr, publisher = {Springer}, series = {Lecture Notes in Computer Science}, site = {Stavanger, Norway}, url = {https://arxiv.org/abs/2201.09992} }
数据集概述
数据集描述
- 名称: HC4
- 类型: 跨语言信息检索测试集合
- 语言: 中文、波斯语、俄语
- 许可证: ODC-BY
- 大小: 1M<n<10M
- 来源: 扩展自c4数据集
- 任务类别: 文本检索
- 任务ID: 文档检索
数据集结构
数据实例
| 分割 | 文档数量 |
|---|---|
fas (波斯语) |
486K |
rus (俄语) |
4.7M |
zho (中文) |
646K |
数据字段
id: 文档的唯一标识符cc_file: 来自Common Crawl的源文件time: 从文章中提取的日期/时间title: 从文章中提取的标题text: 提取的文章主体url: 源URL
数据集使用
使用🤗 Datasets加载数据集的示例代码:
python from datasets import load_dataset
dataset = load_dataset(neuclir/hc4) dataset[fas] # 波斯语文档 dataset[rus] # 俄语文档 dataset[zho] # 中语文档
引用信息
@article{Lawrie2022HC4, author = {Dawn Lawrie and James Mayfield and Douglas W. Oard and Eugene Yang}, title = {HC4: A New Suite of Test Collections for Ad Hoc CLIR}, booktitle = {{Advances in Information Retrieval. 44th European Conference on IR Research (ECIR 2022)}, year = {2022}, month = apr, publisher = {Springer}, series = {Lecture Notes in Computer Science}, site = {Stavanger, Norway}, url = {https://arxiv.org/abs/2201.09992} }
