lkcoffee/sakizaya-corpus

---
language:
- szy
- zh
license: cc-by-nc-sa-4.0
task_categories:
- translation
- text-generation
pretty_name: Sakizaya Language Corpus
size_categories:
- 10K<n<100K
---

# Sakizaya Language Corpus

## Dataset Description

The Sakizaya Language Corpus is a parallel text and lexicon dataset for the Sakizaya language (ISO 639-3: `szy`), an Austronesian language indigenous to the Hualien coastal plain of Taiwan. Sakizaya is a critically endangered language with an estimated few hundred fluent speakers remaining.

This dataset contains:
- **59,512 parallel sentence pairs** (Sakizaya ↔ Traditional Chinese)
- **6,173 lexicon entries** with part-of-speech, definitions, confidence scores, and usage examples

The corpus is intended to support NLP research, machine translation development, and language documentation efforts for low-resource indigenous languages.

## Languages

| Code | Language | Script | Notes |
|------|----------|--------|-------|
| `szy` | Sakizaya | Latin (romanized) | Austronesian; spoken in Hualien, Taiwan |
| `zh` | Traditional Chinese | Han (Traditional) | Used as pivot/target language |

## Data Sources

- **ILRDF (Indigenous Languages Research and Development Foundation, Taiwan)** — official Sakizaya language materials and educational texts
- **Wikipedia (Sakizaya-language articles)** — encyclopedic content scraped and aligned
- **Local documents** — supplementary texts from community and educational sources

## Dataset Structure

### `parallel.csv`

Sentence-level parallel corpus aligned between Sakizaya and Traditional Chinese.

| Column | Type | Description |
|--------|------|-------------|
| `id` | integer | Unique row identifier |
| `article_title` | string | Source article or document title |
| `szy` | string | Sakizaya text (romanized, length > 10 chars) |
| `zh` | string | Traditional Chinese translation (length > 3 chars) |
| `source` | string | Data source identifier (e.g., `ilrdf`, `wiki`) |

### `lexicon.csv`

Word-level lexicon with grammatical and semantic annotations.

| Column | Type | Description |
|--------|------|-------------|
| `word` | string | Sakizaya word form (romanized) |
| `pos` | string | Part of speech (e.g., `n`, `v`, `adj`) |
| `meaning_zh` | string | Meaning in Traditional Chinese |
| `confidence` | float | Confidence score of the entry (0.0–1.0) |
| `source` | string | Data source identifier |
| `example_szy` | string | Example sentence in Sakizaya (may be null) |
| `example_zh` | string | Example sentence in Traditional Chinese (may be null) |

## Usage Example

```python
import pandas as pd

# Load parallel corpus
parallel = pd.read_csv("parallel.csv", encoding="utf-8-sig")
print(f"Parallel pairs: {len(parallel):,}")
print(parallel.head())

# Load lexicon
lexicon = pd.read_csv("lexicon.csv", encoding="utf-8-sig")
print(f"Lexicon entries: {len(lexicon):,}")
print(lexicon[lexicon["pos"] == "n"].head())

# Filter by source
ilrdf_pairs = parallel[parallel["source"] == "ilrdf"]
print(f"ILRDF sentences: {len(ilrdf_pairs):,}")
```

## Citation

If you use this dataset in your research, please cite:

```bibtex
@dataset{sakizaya_corpus_2026,
title = {Sakizaya Language Corpus: Parallel Text and Lexicon},
year = {2026},
language = {szy, zh},
license = {CC BY-NC-SA 4.0},
note = {Sakizaya--Traditional Chinese parallel corpus for NLP and language documentation}
}
```

## License

This dataset is released under the [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.

- **Attribution** — You must give appropriate credit and indicate if changes were made.
- **NonCommercial** — You may not use the material for commercial purposes.
- **ShareAlike** — If you remix or build upon the material, you must distribute your contributions under the same license.

Source materials from ILRDF are used in accordance with their open educational licensing terms.