indic-diarbench

# Indic DiarBench A multilingual joint diarization and ASR benchmark for Indian languages, spanning all **22 scheduled languages** of India with approximately **108 hours** of natural multi-speaker audio. **Paper:** [Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages](https://arxiv.org/abs/2607.23808) (Interspeech 2026) ## Dataset Summary Indic DiarBench is a conversational speech benchmark designed to evaluate speaker-attributed ASR in realistic multi-speaker settings for Indian languages. All annotations are human-corrected with time-aligned, speaker-attributed transcriptions. The dataset captures conversational nuances prevalent in Indian speech, such as English code-mixing, dialectal variation, and frequent speaker overlap. | Attribute | Value | |---|---| | **Total samples** | 1,164 | | **Total duration** | ~108 hours | | **Languages** | 22 scheduled Indian languages | | **Language families** | 4 (Indo-Aryan, Dravidian, Sino-Tibetan, Austroasiatic) | | **Unique speakers** | 485 (meetings) + ~750 (in-the-wild) | | **Districts represented** | 189 | | **Average overlap ratio** | 12.8% | ## Recording Conditions The corpus includes three acoustic conditions designed to capture diverse real-world scenarios: | Condition | Label | Hours | Description | |---|---|---|---| | **Near-field** | `Near field` | ~53 | Recorded using one close-proximity microphone per speaker in virtual meetings. Participants joined via an online meeting platform, enabling accurate speaker turns by combining individual microphone streams. Covers all 22 languages. | | **Far-field** | `Far field` | ~27 | Recorded using distant microphones, introducing reverberation, background noise, and variable speaker-to-microphone distances. Covers the top 8 languages by native-speaker population. | | **In-the-wild** | `In the wild` | ~28 | Curated from publicly available YouTube videos to capture unconstrained acoustic environments. Covers the 10 most widely spoken Indian languages. | ## Per-Language Statistics Durations are in hours. NF = near-field, FF = far-field, ITW = in-the-wild. | Language | Family | NF | FF | ITW | Total | Overlap % | |---|---|---|---|---|---|---| | Assamese | Indo-Aryan | 1.5 | – | – | 1.5 | 13.9 | | Bengali | Indo-Aryan | 4.4 | 4.1 | 4.1 | 12.6 | 7.8 | | Bodo | Sino-Tibetan | 1.6 | – | – | 1.6 | 15.2 | | Dogri | Indo-Aryan | 1.4 | – | – | 1.4 | 24.2 | | Gujarati | Indo-Aryan | 4.1 | 4.2 | 2.8 | 11.1 | 7.6 | | Hindi | Indo-Aryan | 4.2 | 4.0 | 2.5 | 10.7 | 16.6 | | Kannada | Dravidian | 3.7 | 1.4 | 3.3 | 8.5 | 15.2 | | Kashmiri | Indo-Aryan | 1.1 | – | – | 1.1 | 21.2 | | Konkani | Indo-Aryan | 1.6 | – | – | 1.6 | 14.0 | | Maithili | Indo-Aryan | 1.3 | – | – | 1.3 | 24.7 | | Malayalam | Dravidian | 1.3 | – | 2.4 | 3.7 | 12.9 | | Manipuri | Sino-Tibetan | 1.5 | – | – | 1.5 | 20.6 | | Marathi | Indo-Aryan | 4.2 | 3.5 | 2.7 | 10.4 | 11.3 | | Nepali | Indo-Aryan | 1.3 | – | – | 1.3 | 22.9 | | Odia | Indo-Aryan | 1.5 | – | 1.6 | 3.1 | 11.1 | | Punjabi | Indo-Aryan | 4.3 | 4.0 | 2.4 | 10.6 | 6.1 | | Sanskrit | Indo-Aryan | 1.6 | – | – | 1.6 | 21.4 | | Santali | Austroasiatic | 1.6 | – | – | 1.6 | 6.5 | | Sindhi | Indo-Aryan | 1.5 | – | – | 1.5 | 16.0 | | Tamil | Dravidian | 4.2 | 2.5 | 3.2 | 10.0 | 12.4 | | Telugu | Dravidian | 3.8 | 3.0 | 2.5 | 9.3 | 20.4 | | Urdu | Indo-Aryan | 1.6 | – | – | 1.6 | 12.5 | | **Total** | **4 families** | **53.2** | **26.8** | **27.6** | **~108** | **12.8** | ## Annotation Pipeline All recordings are annotated using a unified human-in-the-loop pipeline: 1. **Bootstrap Transcription** — Initial transcripts generated using multiple independent ASR systems, presented to annotators as editable drafts. 2. **Human Transcription & Speaker Attribution** — Professional annotators produce time-aligned, speaker-attributed transcriptions. No machine-generated annotation is retained without human validation. 3. **Code-Mixed Transcription** — Annotators produce two transcription formats: native-script (all text in Indic script) and normalized (English words in Roman script, numerals in Arabic digits). 4. **Quality Control** — Dedicated quality checkers (2–3 per language) verify transcription consistency, code-mixing conventions, speaker timestamps, and labels. Overlapping speech segments require multiple review rounds. 5. **Expert Review** — In-house language-specific experts perform final quality checks. ## Dataset Fields | Field | Type | Description | |---|---|---| | `audio` | Audio | Audio waveform (WAV, 16kHz mono) | | `recording_id` | string | Anonymised source-recording identifier (e.g. `hindi_nf_003`). Clips cut from the same source recording share one `recording_id`. | | `language` | string | Full language name (e.g. `Hindi`), consistent across all recording conditions | | `annotated_transcript` | list | Speaker-attributed segments: `{speaker_id, transcript, start_time, end_time}` | | `dataset_type` | string | Recording condition: `Near field`, `Far field`, or `In the wild` | | `sample_id` | string | Unique sample identifier (e.g. `hindi_001`), unique across the whole benchmark | | `num_speakers` | int | Number of distinct speakers in the recording | | `num_segments` | int | Number of transcript segments | | `duration_seconds` | float | Audio duration in seconds | ### Identifying samples `sample_id` is unique across the entire benchmark (all 1,164 samples), so it is the key to use when reporting or joining per-sample results. `recording_id` exists because the 1,164 clips come from only **590 distinct source recordings** — 47 recordings contribute more than one clip, and one contributes 21. Clips from the same recording are **not independent samples**: they share speakers, channel and acoustic conditions. Group by `recording_id` when splitting data or aggregating metrics, otherwise a handful of recordings will dominate the average. `recording_id` is deliberately anonymised and carries no information about the source medium. Its form is `<language>_<condition>_<nnn>`, where condition is `nf` (near-field), `ff` (far-field) or `itw` (in-the-wild). ## Usage ```python from datasets import load_dataset # Load a specific language ds = load_dataset("sarvamai/indic-diarbench", "Hindi", split="test") sample = ds[0] print(f"Sample: {sample['sample_id']}") print(f"Recording: {sample['recording_id']}") print(f"Language: {sample['language']}") print(f"Duration: {sample['duration_seconds']:.1f}s") print(f"Speakers: {sample['num_speakers']}") print(f"Condition: {sample['dataset_type']}") # Access speaker-attributed transcript for seg in sample['annotated_transcript'][:5]: print(f" [{seg['start_time']:.1f}-{seg['end_time']:.1f}] {seg['speaker_id']}: {seg['transcript']}") ``` ## Baseline Results Duration-weighted aggregate metrics across all three acoustic conditions: | Category | Model | DER (%) | cpWER (%) | WDER (%) | |---|---|---|---|---| | Indic-specialized | Sarvam | **16.0** | **38.8** | **33.1** | | Commercial APIs | AWS Transcribe | 23.5 | 43.7 | 34.3 | | | ElevenLabs Scribe | 35.0 | 58.3 | 40.7 | | | Azure STT | 34.8 | 60.8 | 39.5 | | | Deepgram Nova-3 | 32.0 | 63.2 | 39.3 | | | AssemblyAI | 40.5 | 88.6 | 43.7 | | Multimodal LLMs | GPT-4o | 36.2 | 83.1 | 40.4 | | | Gemini 3 Pro | 74.0 | 58.9 | 33.0 | ## Citation If you use this dataset, please cite: ```bibtex @misc{mehendale2026indicdiarbenchmultilingualjoint, title={Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages}, author={Deovrat Mehendale and Aditya Mehndiratta and Dhruv Rathi and Kaushal Bhogale and Mitesh M. Khapra}, year={2026}, eprint={2607.23808}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2607.23808}, } ``` ## License This dataset is released under the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.