BramVanroy/finemath-4plus-seqlen36k

---
dataset_info:
features:
- name: url
dtype: string
- name: text
dtype: string
- name: token_count
dtype: int32
- name: score
dtype: float64
- name: language_score
dtype: float64
splits:
- name: train
num_bytes: 14821054043
num_examples: 2849400
download_size: 7113010445
dataset_size: 14821054043
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
---

An extraction of https://huggingface.co/datasets/HuggingFaceTB/finemath/viewer/finemath-4plus where:

- only sequences of at most 36,000 tokens are kept;
- only samples that score 4 or more are kept - but calculated differently than in FineMath 4plus. They first round to the nearest integer (so 3.5001 = 4), whereas here I filter directly on the float score where only 4.0+ is actually considered 4plus. This leads to a sharp drop off in number of samples but that is fine for what I need.

Code:

```python
from datasets import load_dataset

ds = load_dataset("HuggingFaceTB/finemath", "finemath-4plus", split="train").select_columns(["url", "text", "token_count", "score", "language_score"]).sort("token_count", reverse=True)

ds = ds.filter(
lambda num_tokens, score: num_tokens <= 36000 and score >= 4.0,
num_proc=36,
input_columns=["token_count", "score"],
)

ds.push_to_hub("BramVanroy/finemath-4plus-seqlen36k")
```