llm-compression

This is the compression corpora dataset used in the paper "Compression Represents Intelligence Linearly".
We find that LLMs’ intelligence – reflected by benchmark scores – almost **linearly** correlates with their ability to compress external text corpora. We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning, and provide the corresponding compression corpora here respectively named cc, python, and arxiv_math.

### Load the data
```python
from datasets import load_dataset
dataset=load_dataset(r"hkust-nlp/llm-compression",name="python")

print(dataset['test'][0])
```
More details on compression evaluation are at our [github page](https://github.com/hkust-nlp/llm-compression-intelligence).

### Citation
```
@misc{huang2024compression,
title={Compression Represents Intelligence Linearly},
author={Yuzhen Huang and Jinghan Zhang and Zifei Shan and Junxian He},
year={2024},
eprint={2404.09937},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

本数据集为论文《Compression Represents Intelligence Linearly》（《压缩可线性表征智能》）中使用的压缩语料库数据集。我们发现，大语言模型（Large Language Model，简称LLM）的智能——以基准测试得分体现——几乎与它们压缩外部文本语料库的能力呈**线性**相关。我们从三大核心能力维度衡量智能：知识与常识、代码编写以及数学推理，并在此提供对应的压缩语料库，分别命名为cc、python与arxiv_math。

### 数据加载
python
from datasets import load_dataset
dataset=load_dataset(r"hkust-nlp/llm-compression",name="python")

print(dataset['test'][0])

有关压缩评估的更多细节可参见我们的[GitHub页面](https://github.com/hkust-nlp/llm-compression-intelligence)。

### 引用信息

@misc{huang2024compression,
title={Compression Represents Intelligence Linearly},
author={Yuzhen Huang and Jinghan Zhang and Zifei Shan and Junxian He},
year={2024},
eprint={2404.09937},
archivePrefix={arXiv},
primaryClass={cs.CL}
}