ezipe/lichess_2021_to_2024_shards

---
license: apache-2.0
---

## Installation
```
pip install datasets numpy zstandard
```

## Usage

i've given up on trying to get this to work natively with huggingface datasets. this is nice because it allows streaming (https://huggingface.co/docs/datasets/en/about_mapstyle_vs_iterable) and has functions like 'map' which easily parallelize operations over the dataset. maybe i'll try to get this working in the future, but for now it gets stuck as the download and extracted zstd files are not decompressible for some reason, and it has to rewrite the entire dataset into arrow first.

<!-- ```
from datasets import load_dataset

dataset = load_dataset("ezipe/lichess_2023_janoct_shards")

``` -->

or more slowly,

```
kconda install git-lfs
git lfs install
git clone https://huggingface.co/datasets/ezipe/lichess_2023_janoct_shards/
```

原始信息汇总

数据集概述

许可证

• 许可证类型：Apache-2.0

安装要求

• 安装依赖：datasets, numpy, zstandard

数据集加载

• 通过Hugging Face的datasets库加载数据集，但由于技术问题，目前无法直接使用。
• 替代方法：通过Git LFS克隆数据集到本地。