willchow66/mmmlu-bias-experiments

---
license: mit
task_categories:
- question-answering
language:
- en
- zh
multilinguality:
- multilingual
size_categories:
- 100K<n<1M
tags:
- llm-bias
- language-bias
- position-bias
- multilingual-evaluation
configs:
- config_name: exp1
data_files:
- split: train
path: exp1/train.json
- config_name: exp1_pos
data_files:
- split: train
path: exp1_pos/train.json
- config_name: exp2
data_files:
- split: train
path: exp2/train.json
- config_name: exp2_pos
data_files:
- split: train
path: exp2_pos/train.json
- config_name: exp3
data_files:
- split: train
path: exp3/train.json
- config_name: exp3_pos
data_files:
- split: train
path: exp3_pos/train.json
- config_name: exp4
data_files:
- split: train
path: exp4/train.json
- config_name: exp4_pos
data_files:
- split: train
path: exp4_pos/train.json
- config_name: exp5
data_files:
- split: train
path: exp5/train.json
- config_name: exp6
data_files:
- split: train
path: exp6/train.json
- config_name: exp7
data_files:
- split: train
path: exp7/train.json
- config_name: exp8
data_files:
- split: train
path: exp8/train.json
---

# MMMLU Bias Experiments Dataset

## Dataset Description

This dataset contains **12 carefully designed experiments** to measure language bias and position bias in Large Language Models (LLMs) using multilingual pairwise judgments.

### Key Features

- **12 Experiments**: 8 original + 4 position-swapped experiments
- **11,478 samples** per experiment (137,736 total test cases)
- **Deterministic wrong answers**: Uses fixed rule `wrong_index = (correct_index + 1) % 4`
- **Perfect correspondence**: Wrong answers are consistent across paired experiments
- **Position bias control**: Position-swapped experiments enable separation of language bias and position bias

### Experiment Design

| Exp | Question Lang | Answer 1 | Answer 2 | Correct | Test Target |
|-----|--------------|----------|----------|---------|-------------|
| **exp1** | English | ✓ English | ✗ Chinese | Answer 1 | Inter-lang: EN context baseline |
| **exp1_pos** | English | ✗ Chinese | ✓ English | Answer 2 | Position swap (EN in pos 2) |
| **exp2** | English | ✗ English | ✓ Chinese | Answer 2 | Inter-lang: EN context test |
| **exp2_pos** | English | ✓ Chinese | ✗ English | Answer 1 | Position swap (CN in pos 1) |
| **exp3** | Chinese | ✓ English | ✗ Chinese | Answer 1 | Inter-lang: CN context baseline |
| **exp3_pos** | Chinese | ✗ Chinese | ✓ English | Answer 2 | Position swap (EN in pos 2) |
| **exp4** | Chinese | ✗ English | ✓ Chinese | Answer 2 | Inter-lang: CN context test |
| **exp4_pos** | Chinese | ✓ Chinese | ✗ English | Answer 1 | Position swap (CN in pos 1) |
| **exp5** | English | ✓ English | ✗ English | Answer 1 | Same-lang: EN baseline |
| **exp6** | Chinese | ✓ Chinese | ✗ Chinese | Answer 1 | Same-lang: CN baseline |
| **exp7** | Chinese | ✓ English | ✗ English | Answer 1 | Robustness: CN Q + EN answers |
| **exp8** | English | ✓ Chinese | ✗ Chinese | Answer 1 | Robustness: EN Q + CN answers |

### Bias Metrics

#### 1. Language Bias (Observed)
```
Observed Bias = Error(wrong answer is CN) - Error(wrong answer is EN)
```
- Positive (+): Model prefers English answers
- Negative (-): Model prefers Chinese answers
- Near 0: No language preference

#### 2. Position Bias
```
Position Bias = [Error(Exp1) - Error(Exp1_pos)] + [Error(Exp2) - Error(Exp2_pos)] / 2
```
- Positive (+): Model prefers Answer 1 position
- Negative (-): Model prefers Answer 2 position
- Near 0: No position preference

#### 3. Pure Language Bias
```
Pure Language Bias ≈ Observed Bias (after position correction)
```

Through position-swapped experiments, we can estimate pure language preference after removing position effects.

### Data Fields

Each example contains:
- `question`: Question text (English or Chinese)
- `answer_1`: First answer choice
- `answer_2`: Second answer choice
- `answer`: Correct answer (matches either answer_1 or answer_2)
- `subject`: Subject category (55 subjects total)
- `split`: Always "test"
- `original_index`: Index in the original MMLU dataset
- `experiment`: Experiment name (exp1, exp1_pos, ..., exp8)

### Dataset Viewer

On Hugging Face, you can use the **Config dropdown** in Dataset Viewer to browse different experiments:

- Select `exp1`, `exp1_pos`, `exp2`, etc. from the dropdown menu
- View samples directly in the browser
- Compare different experiments side-by-side

### Usage Example

```python
from datasets import load_dataset

# Method 1: Load a specific experiment using config name
dataset = load_dataset("willchow66/mmmlu-bias-experiments", "exp1")
print(f"Exp1 samples: {len(dataset['train'])}") # 11,478

# Method 2: Load all configs
from datasets import load_dataset

all_configs = [
"exp1", "exp1_pos", "exp2", "exp2_pos",
"exp3", "exp3_pos", "exp4", "exp4_pos",
"exp5", "exp6", "exp7", "exp8"
]

datasets = {}
for config in all_configs:
datasets[config] = load_dataset("willchow66/mmmlu-bias-experiments", config)
print(f"{config}: {len(datasets[config]['train'])} samples")

# Load position-swapped pair
exp1 = load_dataset("willchow66/mmmlu-bias-experiments", "exp1")
exp1_pos = load_dataset("willchow66/mmmlu-bias-experiments", "exp1_pos")

# Verify correspondence
sample_idx = 0
exp1_sample = exp1['train'][sample_idx]
exp1_pos_sample = exp1_pos['train'][sample_idx]

# Same question and wrong answer
assert exp1_sample['question'] == exp1_pos_sample['question']
assert exp1_sample['answer_2'] == exp1_pos_sample['answer_1'] # Wrong answer swapped positions
```

### Available Configs

| Config | Description | Samples |
|--------|-------------|---------|
| `exp1` | EN question, ✓EN vs ✗CN, Answer 1 correct | 11,478 |
| `exp1_pos` | EN question, ✗CN vs ✓EN, Answer 2 correct (position swap) | 11,478 |
| `exp2` | EN question, ✗EN vs ✓CN, Answer 2 correct | 11,478 |
| `exp2_pos` | EN question, ✓CN vs ✗EN, Answer 1 correct (position swap) | 11,478 |
| `exp3` | CN question, ✓EN vs ✗CN, Answer 1 correct | 11,478 |
| `exp3_pos` | CN question, ✗CN vs ✓EN, Answer 2 correct (position swap) | 11,478 |
| `exp4` | CN question, ✗EN vs ✓CN, Answer 2 correct | 11,478 |
| `exp4_pos` | CN question, ✓CN vs ✗EN, Answer 1 correct (position swap) | 11,478 |
| `exp5` | EN question, ✓EN vs ✗EN, Answer 1 correct | 11,478 |
| `exp6` | CN question, ✓CN vs ✗CN, Answer 1 correct | 11,478 |
| `exp7` | CN question, ✓EN vs ✗EN, Answer 1 correct | 11,478 |
| `exp8` | EN question, ✓CN vs ✗CN, Answer 1 correct | 11,478 |

### Dataset Statistics

- **Total experiments**: 12
- **Samples per experiment**: 11,478
- **Total test cases**: 137,736
- **Subjects**: 55 (STEM, Humanities, Social Sciences, Professional)
- **Languages**: English, Chinese (Simplified)
- **Data source**: [MMMLU Intersection Filtered](https://huggingface.co/datasets/willchow66/mmmlu-intersection-filtered)

### Data Quality

✅ **Perfect Alignment**:
- All paired experiments (e.g., exp1 & exp1_pos) have 100% correspondence
- Wrong answers use deterministic rule, not random selection
- Enables accurate position bias measurement

✅ **Language Character Filtering**:
- All Chinese answers contain Chinese characters
- All English answers contain English text
- No mixed-language contamination

✅ **Subject Coverage**:
- 55 subjects across 14 categories
- Enables fine-grained bias analysis by subject/category
- Sample size ranges from 10 to 1,520 per subject

### Use Cases

1. **Language Bias Detection**: Measure if LLMs prefer answers in certain languages
2. **Position Bias Analysis**: Detect if models favor answers in specific positions
3. **Cross-lingual Consistency**: Test if model judgments remain consistent across languages
4. **Model Comparison**: Compare bias patterns across different LLM architectures/sizes
5. **Bias Evolution**: Track how bias changes across model versions

### Citation

If you use this dataset, please cite:

```bibtex
@dataset{mmmlu_bias_experiments_2025,
author = {Zhou, Xin},
title = {MMMLU Bias Experiments: Multilingual Pairwise Judgment Dataset for LLM Bias Detection},
year = {2025},
publisher = {Hugging Face},
url = {https://huggingface.co/datasets/willchow66/mmmlu-bias-experiments}
}
```

### Related Datasets

- [MMMLU Intersection Filtered](https://huggingface.co/datasets/willchow66/mmmlu-intersection-filtered) - Source dataset (15 languages)
- [MMMLU](https://huggingface.co/datasets/openai/MMMLU) - Original multilingual dataset
- [MMLU](https://huggingface.co/datasets/cais/mmlu) - Original English dataset

### License

MIT License

### Acknowledgments

This dataset is derived from MMMLU (OpenAI) and MMLU (CAIS), with additional processing for bias research.