sartifyllc/kiswahili-asr-challenge-eval

---
license: cc-by-sa-4.0
task_categories:
- automatic-speech-recognition
language:
- sw
tags:
- swahili
- kiswahili
- asr
- speech-to-text
- stt
- on-device
- edge-ai
- low-resource
- africa
- zindi
- sartify
- ai-for-good
- itu
pretty_name: "Swahili ASR Zindi Evaluation Dataset"
size_categories:
- 1K<n<10K
---

# Swahili ASR Zindi Evaluation Dataset

**Presented by [Sartify](https://sartify.com) in partnership with [AI for Good (ITU)](https://aiforgood.itu.int/) on the [Zindi](https://zindi.africa) platform.**

## Dataset Summary

This is the official evaluation (test) dataset for the **"Your Voice, Your Device, Your Language"** challenge — a Zindi competition focused on building lightweight, on-device Kiswahili Automatic Speech Recognition (ASR) systems.

The dataset contains Kiswahili audio samples intended for evaluating speech-to-text models under real-world, conversational conditions. Participants are tasked with transcribing these audio files into accurate Kiswahili text, with a focus on solutions that are fast, secure, and fully offline-capable.

> 🌍 Kiswahili is spoken by over **200 million people** across East and Central Africa. This dataset supports the development of inclusive voice technologies for communities too often left behind by mainstream AI.

## Dataset Details

| Property | Value |
|---|---|
| **Language** | Kiswahili (Swahili) — `sw` |
| **Task** | Automatic Speech Recognition (ASR) |
| **Audio Format** | `.wav` |
| **License** | CC-BY-SA 4.0 |
| **Source** | [Zindi Challenge Page](https://zindi.africa/competitions/your-voice-your-device-your-language-challenge) |
| **Challenge Duration** | 22 July 2025 – 22 September 2025 |

## Dataset Structure

### Files

- **Audio files**: `.wav` files named with UUIDs (e.g., `451f6d89-9b85-46c3-ad8d-bfcb1c9a4e8f.wav`)
- **Submission format**: CSV with two columns — `filename` and `text`

### Example

| filename | text |
|---|---|
| `451f6d89-9b85-46c3-ad8d-bfcb1c9a4e8f.wav` | mwanadamu Biblia |
| `507e10f8-0b2b-4bc0-9b69-94f96d907fb6.wav` | upana baina |

## Intended Use

This dataset is designed to:

- **Evaluate** Kiswahili ASR models for accuracy using Word Error Rate (WER)
- **Benchmark** on-device and edge-deployable speech recognition systems
- **Advance** inclusive language technology for low-resource African languages
- **Support** privacy-preserving, offline voice interfaces for underserved communities

### Target Deployment Constraints

Solutions built against this evaluation set are expected to meet the following real-world constraints:

- Single **NVIDIA T4 GPU** with ≤16 GB RAM
- Real-time inference capability (measured by Real-Time Factor)
- Bonus: Full pipeline within **4 GB memory** for mobile/edge deployment

## Evaluation

### Phase One — Error Metric

The primary metric is **Word Error Rate (WER)**.

### Phase Two — Performance Evaluation (Top 10)

| Criterion | Weight |
|---|---|
| WER Performance | 85% |
| Real-Time Factor (RTFx) on NVIDIA T4 | 5% |
| Peak GPU Memory (≤16 GB) | 5% |
| Documentation & Reproducibility | 5% |

## Challenge Context

### Why This Matters

Most cutting-edge speech-to-text systems depend on cloud services, large GPUs, and high-bandwidth internet — none of which are guaranteed in many Kiswahili-speaking communities. Beyond infrastructure barriers, cloud dependence introduces serious privacy risks for sensitive voice data related to health, finances, or legal matters.

This challenge and dataset aim to catalyze the development of **robust, on-device STT models** that are fast, secure, and fully offline.

### Extension Opportunities

Participants and researchers are encouraged to extend this work into:

- **Text-to-Speech (TTS)** systems for Kiswahili
- Full **voice assistant** prototypes
- Real-world applications in **healthcare**, **education**, and **public services**

## Organizations

### Sartify

[Sartify](https://sartify.com) is an African-led AI company focused on inclusive language technologies for low-resource communities. Their portfolio includes core engines — **PAWA-AI** and the **Kinong'ono ASR model** — along with end-user applications such as **DOCIPRO** and **TUTOR-AI**. Driven by the motto *"AI by Africa, for Africa,"* Sartify develops and scales homegrown AI solutions across the continent.

### AI for Good — International Telecommunication Union (ITU)

[AI for Good](https://aiforgood.itu.int/) is organized by the ITU in partnership with 40 UN Sister Agencies. It is the leading action-oriented, global, and inclusive United Nations platform on AI, focused on identifying practical applications of AI to advance the UN Sustainable Development Goals.

### Zindi

[Zindi](https://zindi.africa) is Africa's largest professional network for data scientists, hosting competitions that connect AI talent with real-world challenges across the continent.

## Leaderboard Results

The challenge concluded on **22 September 2025**. Below are the final top 5 rankings based on **Word Error Rate (WER)** — lower is better.

| Rank | User / Team | Affiliation | Public WER | Private WER | Submissions |
|:---:|---|---|:---:|:---:|:---:|
| 🥇 1st | **Abdourahamane_** | Carnegie Mellon University Africa | 0.1822 | **0.1781** | 57 |
| 🥈 2nd | **LAST_HOPE** (Team) | — | 0.1889 | **0.1862** | 131 |
| 🥉 3rd | **keystats** | Mount Kenya University | 0.2116 | **0.2059** | 36 |
| 4th | **Dannie_AI** | — | 0.1915 | 0.2193 | 5 |
| 5th | **moadel2002** | — | 0.2310 | 0.2389 | 103 |

### Prizes

| Place | Winner | Prize |
|:---:|---|:---:|
| 🥇 1st | @Abdourahamane_ | 500 CHF |
| 🥈 2nd | Team LAST_HOPE | 300 CHF |
| 🥉 3rd | @keystats | 200 CHF |

## Citation

If you use this dataset in your research, please cite:

```bibtex
@dataset{sartify_swahili_asr_eval_2025,
title = {Swahili ASR Zindi Evaluation Dataset},
author = {Nyaki, Elizabeth and Innocent Nyankana and Mollel, Michael},
year = {2025},
publisher = {Pawa-AI},
url = {https://huggingface.co/datasets/SartifyLLC/swahili-asr-zindi-eval},
note = {Evaluation dataset for the "Your Voice, Your Device, Your Language" Zindi Challenge}
}
```

## License

This dataset is released under the [**Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA 4.0)**](https://creativecommons.org/licenses/by-sa/4.0/) license.

## Contact

- **Sartify**: [https://sartify.com](https://sartify.com)
- **Zindi Challenge Page**: [Your Voice, Your Device, Your Language Challenge](https://zindi.africa/competitions/your-voice-your-device-your-language-challenge)