BorisTM/bak-rus-parallel

---
dataset_info:
features:
- name: bak
dtype: string
- name: rus
dtype: string
- name: static_sim
dtype: float32
- name: source
dtype: string
- name: length_bak
dtype: int64
- name: length_rus
dtype: int64
splits:
- name: train
num_bytes: 3839636684
num_examples: 9768889
download_size: 1915872020
dataset_size: 3839636684
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
task_categories:
- translation
language:
- ru
- ba
size_categories:
- 1M<n<10M
---

# Russian-Bashkir Parallel Corpus

A large-scale parallel corpus for Russian-Bashkir machine translation, containing 9.7M sentence pairs filtered by semantic similarity.

## Dataset Details

- **Languages**: Russian (`rus`) ↔ Bashkir (`bak`)
- **Size**: 9,768,889 translation pairs
- **Repository**: [`BorisTM/bak-rus-parallel`](https://huggingface.co/datasets/BorisTM/bak-rus-parallel)

## Usage

```python
from datasets import load_dataset

dataset = load_dataset("BorisTM/bak-rus-parallel")
```

## Dataset Fields

| Field | Type | Description |
|-------|------|-------------|
| `rus` | string | Russian sentence |
| `bak` | string | Bashkir sentence |
| `static_sim` | float32 | Semantic similarity score (cosine similarity, 0-1) |
| `source` | string | Data source: `"real"` or `"synth"` |
| `length_rus` | int64 | Character length of Russian sentence |
| `length_bak` | int64 | Character length of Bashkir sentence |

## Dataset Composition

The corpus combines two sources:

### Real Data (`source="real"`)
Authentic human-created translations from the following parallel corpora:

| Source | Description |
|--------|-------------|
| [`AigizK/bashkir-russian-parallel-corpora`](https://huggingface.co/datasets/AigizK/bashkir-russian-parallel-corpora) | Bashkir-Russian parallel corpus |
| **TIL-MT Corpus** | Turkic Interlingua parallel corpus (ba-ru) |
| **NLLB** (OPUS) | NLLB ba-ru parallel corpus from [opus.nlpl.eu](https://opus.nlpl.eu) |
| **Wikimedia** (OPUS) | Wikimedia ba-ru parallel corpus from [opus.nlpl.eu](https://opus.nlpl.eu) |

### Synthetic Data (`source="synth"`)
Generated via back-translation using the [`facebook/nllb-200-distilled-600M`](https://huggingface.co/facebook/nllb-200-distilled-600M) model on the following raw Bashkir text corpora:

| Source | Description |
|--------|-------------|
| [`omarkamali/wikipedia-monthly`](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) | Bashkir Wikipedia (2025-10-01 dump) |
| [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | FineWeb-2 bak_Cyrl split |
| [`cis-lmu/Glot500`](https://huggingface.co/datasets/cis-lmu/Glot500) | Glot500 bak_Cyrl split |
| **HPOLT 3.0** | High-quality multilingual web text (bak_Cyrl) |

## Filtering

All sentence pairs are filtered by the following criteria:

- **Semantic similarity** > 0.25 (computed using [`BorisTM/static_rus_bak`](https://huggingface.co/BorisTM/static_rus_bak) embeddings)
- **Sentence length**: <512 for both languages

## Statistics

| Source | Count |
|--------|-------|
| Real | ~3.0M pairs |
| Synthetic | ~6.8M pairs |
| **Total** | **9.7M pairs** |

## License

Please refer to the original source licenses for specific data components.

## Citation

If you use this dataset, please cite:

```bibtex
@dataset{bak_rus_parallel,
title = {Russian-Bashkir Parallel Corpus},
author = {BorisTM},
year = {2025},
url = {https://huggingface.co/datasets/BorisTM/bak-rus-parallel}
}
```