vlsp-2023-vllm/mmlu

---
configs:
- config_name: default
data_files:
- split: validation
path: data/validation-*
- split: dev
path: data/dev-*
- split: test
path: data/test-*
dataset_info:
features:
- name: id
dtype: string
- name: answer
dtype: int64
- name: question
dtype: string
- name: choices
sequence: string
splits:
- name: validation
num_bytes: 890402
num_examples: 1456
- name: dev
num_bytes: 140819
num_examples: 271
- name: test
num_bytes: 7615124
num_examples: 13062
download_size: 4415183
dataset_size: 8646345
---
References: https://huggingface.co/datasets/cais/mmlu

# MMLU (Vietnamese translation version)

## Install

To install `lm-eval` from the github repository main branch, run:

```bash
git clone https://github.com/hieunguyen1053/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
```

## Basic Usage

> **Note**: When reporting results from eval harness, please include the task versions (shown in `results["versions"]`) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.

### Hugging Face `transformers`

To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. vlsp-2023-vllm/hoa-1b4) on `mmlu` you can use the following command:

```bash
python main.py \
--model hf-causal \
--model_args pretrained=vlsp-2023-vllm/hoa-1b4 \
--tasks mmlu_vi \
--device cuda:0
```

Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

```bash
python main.py \
--model hf-causal \
--model_args pretrained=vlsp-2023-vllm/hoa-1b4,revision=step100000,dtype="float" \
--tasks mmlu_vi \
--device cuda:0
```

To evaluate models that are loaded via `AutoSeq2SeqLM` in Huggingface, you instead use `hf-seq2seq`. *To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`*

> **Warning**: Choosing the wrong model may result in erroneous outputs despite not erroring.

This dataset is a multiple-choice question answering dataset, consisting of three main parts: validation, dev, and test. Each part has its corresponding file paths, number of bytes, and number of examples. The dataset features include id (string type), answer (int64 type), question (string type), and choices (sequence of strings). The download size of the dataset is 4415183 bytes, and the actual size is 8646345 bytes.

原始信息汇总

MMLU (Vietnamese translation version)

数据集配置

默认配置

• 数据文件:
  • 验证集: data/validation-*
  • 开发集: data/dev-*
  • 测试集: data/test-*

数据集信息

• 特征:

  • id: 字符串类型
  • answer: 64位整数类型
  • question: 字符串类型
  • choices: 字符串序列

• 数据分割:

  • 验证集:
    • 字节数: 890402
    • 样本数: 1456
  • 开发集:
    • 字节数: 140819
    • 样本数: 271
  • 测试集:
    • 字节数: 7615124
    • 样本数: 13062

• 下载大小: 4415183 字节

• 数据集大小: 8646345 字节