BrunoHays/fleurs_code_switching_test

---
dataset_info:
features:
- name: audio
dtype:
audio:
sampling_rate: 16000
- name: transcription
dtype: string
- name: transcription_tagged
dtype: string
- name: duration_sec
dtype: float64
- name: languages
list: string
- name: seed
dtype: int64
splits:
- name: test
num_bytes: 9824819984
num_examples: 1000
download_size: 9822674863
dataset_size: 9824819984
configs:
- config_name: default
data_files:
- split: test
path: data/test-*
---

# FLEURS Code-Switching Evaluation Set

## Dataset Summary
This dataset is a synthetic code-switching evaluation set built from the `google/fleurs` corpus.
Each sample is a single long-form audio sequence (minimum 5 minutes by default) composed by concatenating short utterances from multiple languages.

The goal is to provide a controlled benchmark for testing ASR robustness when language switches happen frequently inside one recording.

## How The Dataset Was Curated
- **Source data:** `google/fleurs` Parquet files loaded per language/split (default split: `test`).
- **Languages used:** `en`, `fr`, `es`, `de`, `ru`, `it`, `pt`, `nl`.
- **Per-sample language mix:** a random subset of 2 to 8 languages.
- **Coverage constraint:** each selected language appears at least once in the sample.
- **Construction rule:** utterances are randomly sampled and concatenated until sample duration reaches at least 300 seconds (default).
- **Audio normalization:** utterances are decoded to a common sampling rate (`--target-sr`, default `16kHz`) and converted to mono when needed.

## Columns
- `id`: unique sample index.
- `audio`: concatenated waveform and sampling rate.
- `transcription`: plain concatenation of chunk transcripts.
- `transcription_tagged`: transcript with inline language and timing markers for each chunk, formatted as
`<lang><start:SS.ss>text<end:SS.ss>`.
- `duration_sec`: final sample duration in seconds.
- `languages`: languages selected for that sample.
- `seed`: per-sample random seed.

## Limitations
- Code-switching is synthetic (concatenative), not natural conversational switching.
- Prosody, speaker continuity, and discourse-level transition cues are not preserved across joins.