taeminlee/CLIcK

---
task_categories:
- multiple-choice
language:
- ko
tags:
- Culture
- Language
size_categories:
- 1K<n<10K
configs:
- config_name: KL_Grammar
data_files:
- path:
- Dataset/Language/Grammar/Grammar_CSAT.json
- Dataset/Language/Grammar/Grammar_TOPIK.json
- Dataset/Language/Grammar/Grammar_Kedu.json
split: test
- config_name: KL_Textual
data_files:
- path:
- Dataset/Language/Textual/Textual_TOPIK.json
- Dataset/Language/Textual/Textual_CSAT.json
split: test
- config_name: KL_Functional
data_files:
- path:
- Dataset/Language/Functional/Functional_Kedu.json
- Dataset/Language/Functional/Functional_PSE.json
- Dataset/Language/Functional/Functional_CSAT.json
split: test
- config_name: KC_Law
data_files:
- path:
- Dataset/Culture/Korean Law/Law_KIIP.json
- Dataset/Culture/Korean Law/Law_PSAT.json
split: test
- config_name: KC_Popular
data_files:
- path:
- Dataset/Culture/Korean Popular/Popular_Kedu.json
- Dataset/Culture/Korean Popular/Popular_KIIP.json
split: test
- config_name: KC_Politics
data_files:
- path:
- Dataset/Culture/Korean Politics/Politics_Kedu.json
- Dataset/Culture/Korean Politics/Politics_KIIP.json
split: test
- config_name: KC_Geography
data_files:
- path:
- Dataset/Culture/Korean Geography/Geography_KIIP.json
- Dataset/Culture/Korean Geography/Geography_Kedu.json
- Dataset/Culture/Korean Geography/Geography_CSAT.json
split: test
- config_name: KC_Economy
data_files:
- path:
- Dataset/Culture/Korean Economy/Economy_KIIP.json
- Dataset/Culture/Korean Economy/Economy_Kedu.json
split: test
- config_name: KC_History
data_files:
- path:
- Dataset/Culture/Korean History/History_Kedu.json
- Dataset/Culture/Korean History/History_PSE.json
- Dataset/Culture/Korean History/History_KHB.json
split: test
- config_name: KC_Society
data_files:
- path:
- Dataset/Culture/Korean Society/Society_Kedu.json
- Dataset/Culture/Korean Society/Society_KIIP.json
split: test
- config_name: KC_Tradition
data_files:
- path:
- Dataset/Culture/Korean Tradition/Tradition_Kedu.json
- Dataset/Culture/Korean Tradition/Tradition_KIIP.json
split: test
---

## This dataset is the same as https://huggingface.co/datasets/EunsuKim/CLIcK. This dataset has been subdivided for simplified viewing and evaluation.

<div align="center">
<h1>CLIcK 🇰🇷🧠</h1>
<p>Evaluation of Cultural and Linguistic Intelligence in Korean</p>
<p>
<a href="https://huggingface.co/datasets/your_username/CLIcK"><img src="https://img.shields.io/badge/Dataset-CLIcK-blue" alt="Dataset"></a>
<a href="https://arxiv.org/abs/2403.06412"><img src="https://img.shields.io/badge/Paper-LREC--COLING-green" alt="Paper"></a>
</p>
</div>

## Introduction 🎉

CLIcK (Cultural and Linguistic Intelligence in Korean) is a comprehensive dataset designed to evaluate cultural and linguistic intelligence in the context of Korean language models. In an era where diverse language models are continually emerging, there is a pressing need for robust evaluation datasets, especially for non-English languages like Korean. CLIcK fills this gap by providing a rich, well-categorized dataset focusing on both cultural and linguistic aspects, enabling a nuanced assessment of Korean language models.

## News 📰

- **[LREC-COLING]** Our paper introducing CLIcK has been accepted to LREC-COLING 2024!🎉

## Dataset Description 📊

The CLIcK benchmark comprises two broad categories: Culture and Language, which are further divided into 11 fine-grained subcategories.

### Categories 📂

- **Language** 🗣️
- Textual Knowledge
- Grammatical Knowledge
- Functional Knowledge

- **Culture** 🌍
- Korean Society
- Korean Tradition
- Korean Politics
- Korean Economy
- Korean Law
- Korean History
- Korean Geography
- Korean Popular Culture (K-Pop)

### Construction 🏗️

CLIcK was developed using two human-centric approaches:

1. Reclassification of **official and well-designed exam data** into our defined categories.
2. Generation of questions using ChatGPT, based on **official educational materials** from the Korean Ministry of Justice, followed by our own validation process.

### Structure 🏛️

The dataset is organized as follows, with each subcategory containing relevant JSON files:

```
📦CLIcK
└─ Dataset
├─ Culture
│ ├─ [Each cultural subcategory with associated JSON files]
└─ Language
├─ [Each language subcategory with associated JSON files]
```

### Exam Code Descriptions 📜

- KIIP: Korea Immigration & Integration Program ([Website](www.immigration.go.kr))
- CSAT: College Scholastic Ability Test for Korean ([Website](https://www.suneung.re.kr/))
- Kedu: Test of Teaching Korean as a Foreign Language exams ([Website](https://www.q-net.or.kr/man001.do?gSite=L&gId=36))
- PSE: Public Service Exam for 9th grade
- TOPIK: Test of Proficiency in Korean ([Website](https://www.topik.go.kr/))
- KHB: Korean History Exam Basic ([Website](https://www.historyexam.go.kr/))
- PSAT: Public Service Aptitude Test in Korea

## Results

| Models | Average Accuracy (Korean Culture) | Average Accuracy (Korean Language) |
|-------------------|-----------------------------------|------------------------------------|
| Polyglot-Ko 1.3B | 32.71% | 22.88% |
| Polyglot-Ko 3.8B | 32.90% | 22.38% |
| Polyglot-Ko 5.8B | 33.14% | 23.27% |
| Polyglot-Ko 12.8B | 33.40% | 22.24% |
| KULLM 5.8B | 33.79% | 23.50% |
| KULLM 12.8B | 33.51% | 23.78% |
| KoAlpaca 5.8B | 32.33% | 23.87% |
| KoAlpaca 12.8B | 33.80% | 22.42% |
| LLaMA-Ko 7B | 33.26% | 25.69% |
| LLaMA 7B | 35.44% | 27.17% |
| LLaMA 13B | **36.22%** | **26.71%** |
| GPT-3.5 | 49.30% | 42.32% |
| Claude2 | **51.72%** | **45.39%** |

## Dataset Link 🔗

The CLIcK dataset is available on the Hugging Face Hub: [CLIcK Dataset](https://huggingface.co/datasets/your_username/CLIcK)

## Citation 📝

If you use CLIcK in your research, please cite our paper:

```bibtex
@misc{kim2024click,
title={CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean},
author={Eunsu Kim and Juyoung Suk and Philhoon Oh and Haneul Yoo and James Thorne and Alice Oh},
year={2024},
eprint={2403.06412},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

## Contact 📧

For any questions or inquiries, please contact [[e-mail]](mailto:[e-mail]).

原始信息汇总

数据集概述

任务类别

• 多选题

语言

• 韩语

标签

• 文化
• 语言

大小类别

• 1K<n<10K

配置详情

• KL_Grammar

  • 数据文件路径:
    • Dataset/Language/Grammar/Grammar_CSAT.json
    • Dataset/Language/Grammar/Grammar_TOPIK.json
    • Dataset/Language/Grammar/Grammar_Kedu.json
  • 分割: 测试

• KL_Textual

  • 数据文件路径:
    • Dataset/Language/Textual/Textual_TOPIK.json
    • Dataset/Language/Textual/Textual_CSAT.json
  • 分割: 测试

• KL_Functional

  • 数据文件路径:
    • Dataset/Language/Functional/Functional_Kedu.json
    • Dataset/Language/Functional/Functional_PSE.json
    • Dataset/Language/Functional/Functional_CSAT.json
  • 分割: 测试

• KC_Law

  • 数据文件路径:
    • Dataset/Culture/Korean Law/Law_KIIP.json
    • Dataset/Culture/Korean Law/Law_PSAT.json
  • 分割: 测试

• KC_Popular

  • 数据文件路径:
    • Dataset/Culture/Korean Popular/Popular_Kedu.json
    • Dataset/Culture/Korean Popular/Popular_KIIP.json
  • 分割: 测试

• KC_Politics

  • 数据文件路径:
    • Dataset/Culture/Korean Politics/Politics_Kedu.json
    • Dataset/Culture/Korean Politics/Politics_KIIP.json
  • 分割: 测试

• KC_Geography

  • 数据文件路径:
    • Dataset/Culture/Korean Geography/Geography_KIIP.json
    • Dataset/Culture/Korean Geography/Geography_Kedu.json
    • Dataset/Culture/Korean Geography/Geography_CSAT.json
  • 分割: 测试

• KC_Economy

  • 数据文件路径:
    • Dataset/Culture/Korean Economy/Economy_KIIP.json
    • Dataset/Culture/Korean Economy/Economy_Kedu.json
  • 分割: 测试

• KC_History

  • 数据文件路径:
    • Dataset/Culture/Korean History/History_Kedu.json
    • Dataset/Culture/Korean History/History_PSE.json
    • Dataset/Culture/Korean History/History_KHB.json
  • 分割: 测试

• KC_Society

  • 数据文件路径:
    • Dataset/Culture/Korean Society/Society_Kedu.json
    • Dataset/Culture/Korean Society/Society_KIIP.json
  • 分割: 测试

• KC_Tradition

  • 数据文件路径:
    • Dataset/Culture/Korean Tradition/Tradition_Kedu.json
    • Dataset/Culture/Korean Tradition/Tradition_KIIP.json
  • 分割: 测试

数据集结构

📦CLIcK └─ Dataset ├─ Culture │ ├─ [各文化子类别及其关联的JSON文件] └─ Language ├─ [各语言子类别及其关联的JSON文件]

考试代码描述

• KIIP: 韩国移民与整合计划
• CSAT: 韩国大学学术能力测试
• Kedu: 韩国外语教学考试
• PSE: 公共服务考试第九级
• TOPIK: 韩国语能力测试
• KHB: 韩国历史基础考试
• PSAT: 韩国公共服务能力倾向测试