CoSHE-500

# CoSHE-500 — Conversational Hinglish Speech Evaluation **CoSHE-500** is a 500-clip evaluation set of **conversational Hindi–English code-switched (Hinglish) speech**, with verbatim mixed-script (Devanagari + Latin) reference transcriptions. It is designed to measure real-world code-switch ASR quality — the setting where most Hindi/English models degrade. > **Source.** CoSHE-500 is a curated 500-clip subset of **CoSHE** — > [`soketlabs/CoSHE-Eval`](https://huggingface.co/datasets/soketlabs/CoSHE-Eval) by Soket Labs — > redistributed under its original **CC-BY-NC-4.0** license (attribution, non-commercial). All credit > for the underlying audio and transcriptions goes to the CoSHE / Soket Labs team. We only sampled, > chunked to ~30 s, and tidied the columns. - **Clips:** 500 conversational utterances (~30 s each) - **Split:** `eval` - **Columns:** `audio` (16 kHz mono), `transcription` (verbatim, mixed-script) - **Languages:** Hindi + English, code-switched within utterances - **Script:** Devanagari for Hindi, Latin for English — kept verbatim (no forced transliteration) ## Usage ```python from datasets import load_dataset ds = load_dataset("Trelis/CoSHE-500", split="eval") print(ds[0]["transcription"]) # ds[0]["audio"] -> {"array": np.ndarray, "sampling_rate": 16000} ``` ## Recommended scoring Hinglish WER is sensitive to the normaliser. We recommend a **script-safe Indic normaliser** that: - applies Unicode NFC + Indic normalisation, **keeping** Devanagari matras and nuktas (`ड` ≠ `ड़`), - lowercases Latin, - strips punctuation (including the apostrophe), keeps letters/marks/numbers, - does **not** transliterate and does **not** number-normalise. The default Whisper text normaliser strips Devanagari matras and **inflates** Devanagari WER — do not use it for this benchmark. Report **corpus WER** (sum of edits / sum of reference words), lower is better. > **Note for code-switch scoring:** systems that emit English loanwords in Latin or numbers as digits will > be penalised against these Devanagari references by a translit-blind WER. State your output convention > when reporting. ## Intended use Evaluation / benchmarking of ASR systems on conversational Hinglish. **Not** a training set. Please do not train on CoSHE-500 — keep it held out so it remains a fair public yardstick. ## License & attribution **CC-BY-NC-4.0**, inherited from the source dataset [`soketlabs/CoSHE-Eval`](https://huggingface.co/datasets/soketlabs/CoSHE-Eval) (Soket Labs). Attribution required; **non-commercial** use only. Please cite CoSHE / Soket Labs when you use this benchmark. This repackaging (subset, 30 s chunking, column cleanup) is provided for convenient evaluation; the audio and reference transcriptions remain the property of the original CoSHE authors.