Bonkh/multilingual-benchmark-dataset

---
language:
- multilingual
task_categories:
- text-classification
tags:
- language-identification
- lid
---

# Multilingual benchmark dataset

This dataset contains cleaned text data organized by language, and can be used as a benchmark for evaluating language identification (LID) models.

## Overview

This dataset consists of samples from multiple existing benchmark and multilingual text datasets. The full list of sources and licenses is provided [here](https://docs.google.com/spreadsheets/d/1G12FaSMelNX87dclhm9dE3d5ZRO99B2hoFE1ufB2Zvg/edit?usp=sharing).

- **Languages:** ~1,800 languages
- **Samples:** Over 1M
- **Licenses:** All sources are commercially usable — full license list provided in the sheet above

## Dataset Structure

- Each parquet file corresponds to one language (e.g. `eng.parquet`, `fra.parquet`)
- Columns: `text`, `language`

## Data Cleaning Pipeline

The dataset was carefully processed through the following steps:
1. **Deduplication** — exact duplicate removal across all sources
2. **Filtering** — removal of programming language code, abnormal symbols, and numeric-only samples
3. **Semantic deduplication** — near-duplicate removal using [SemHash](https://github.com/MinishLab/semhash)
4. **Mislabel correction** — noisy label detection and removal using [CleanLab](https://github.com/cleanlab/cleanlab)