IndicCMix

# IndicCMix Most Indic NLP data assumes people write in one script and one language at a time. Real chat looks nothing like that. You get Hindi words in Roman letters, English verbs in the middle of a Tamil sentence, and the same person switching scripts halfway through a paragraph. This dataset is an attempt to cover that actual messiness. For every English sentence, you get three different Indic renderings of it: one code-mixed in the native script, one clean native-script translation, and one romanized version of the kind people type on WhatsApp. Each English sentence also comes with an audio clip, so if you want to do speech-side experiments, the data supports that too. ## What's in the dataset | | | |---|---| | Rows | 1,152,750 | | Languages | 11 Indic + English | | Unique English sentences | ~104,809 | | Audio files | ~104,809 WAVs, ~31 GB total | | Audio source | Synthesized with [F5-TTS](https://github.com/SWivid/F5-TTS) | | Viewer preview | 110 rows with playable audio | ## Fields | Field | What it is | |---|---| | `id` | Row identifier, e.g. `hi_000042` | | `language` | One of `as`, `bn`, `gu`, `hi`, `ka`, `ml`, `mr`, `or`, `pa`, `ta`, `te` | | `english` | The source sentence in English | | `native_script_codemixed` | Indic script with English words left in place, the way people actually code-switch | | `full_native_script` | Full translation into the target language, no English words | | `romanized_casual` | Roman-letter version, WhatsApp style | | `audio` | F5-TTS English speech of the `english` field | ### One row in full (Hindi) ``` id: hi_314203 language: hi english: The Indian festival organizers partnered with a fintech company to introduce digital payment options like Paytm and UPI for ticket bookings and donations. native_script_codemixed: भारतीय festival organizers ने एक fintech company के साथ partnership की है ताकि ticket bookings और donations के लिए Paytm और UPI जैसे digital payment options शुरू कर सकें। full_native_script: भारतीय फेस्टिवल ऑर्गनाइजर्स ने एक फिनटेक कंपनी के साथ पार्टनरशिप की है ताकि टिकट बुकिंग्स और डोनेशन्स के लिए पेयटएम और यूपीआई जैसे डिजिटल पेमेंट ऑप्शन्स शुरू कर सकें। romanized_casual: Bharatiya tyohar aayojakon ne ek fintech company ke saath milkar ticket booking aur donation ke liye Paytm aur UPI jaise digital payment options shuru kiye hain. audio: audio/XX/000630.wav ``` ## Languages and row counts | Code | Language | Rows | |------|------------|---------| | as | Assamese | 104,788 | | bn | Bengali | 104,795 | | gu | Gujarati | 104,796 | | hi | Hindi | 104,800 | | ka | Kannada | 104,798 | | ml | Malayalam | 104,798 | | mr | Marathi | 104,791 | | or | Odia | 104,794 | | pa | Punjabi | 104,797 | | ta | Tamil | 104,797 | | te | Telugu | 104,796 | Each English sentence goes into all 11 languages, so the audio files are shared. Only the Indic text differs between configs. ## How it was made A corpus of roughly 104k English sentences was translated into all 11 Indic languages. `native_script_codemixed` is the base form, with English words kept as-is in Roman letters. `full_native_script` is derived from it by transliterating those English tokens into the target script, so the two are aligned word-for-word and safe to use as parallel data. `romanized_casual` is generated in a separate pass, so it carries the same meaning but is not token-aligned with the other two. The English sentences were then fed to F5-TTS to generate the audio. ## Loading Preview (small, with playable audio): ```python from datasets import load_dataset ds = load_dataset("ai4bharat/IndicCMix", split="train") print(ds[0]) ``` One language, text only (fastest): ```python from datasets import load_dataset ds = load_dataset("ai4bharat/IndicCMix", data_files="hi.parquet", split="train") ``` One language with playable audio: ```python from datasets import load_dataset, Audio ds = load_dataset("ai4bharat/IndicCMix", data_files="hi.parquet", split="train") ds = ds.cast_column("audio", Audio()) ``` Everything at once: ```python from datasets import load_dataset, Audio files = [f"{l}.parquet" for l in ["as","bn","gu","hi","ka","ml","mr","or","pa","ta","te"]] ds = load_dataset("ai4bharat/IndicCMix", data_files=files, split="train") ds = ds.cast_column("audio", Audio()) ``` Offline snapshot: ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="ai4bharat/IndicCMix", repo_type="dataset", local_dir="./IndicCMix", ) ``` ## What you can actually do with it On the text side, the obvious ones: English to code-mixed generation, English to native-script translation, and transliteration between romanized and native Indic. Less obvious but more interesting: training a single model that can switch between all three output styles on command, or probing an LLM to see whether it can code-switch at all. On the speech side, you get English audio paired with Indic text in three forms, which is a strange and useful setup. You can train speech-to-code-mixed or speech-to-romanized models directly. Keep in mind the audio is synthetic (F5-TTS), so models trained only on this will need real recordings before they are useful in production. ## Repo layout ``` ├── data/ │ └── train-00000-of-00001.parquet # Preview with embedded audio ├── as.parquet, bn.parquet, ... # Full per-language files (audio as paths) ├── audio/ │ ├── 00/ ... ff/ # WAVs sharded into 256 subdirs │ │ └── *.wav └── README.md ``` The audio is split into 256 subdirectories by MD5 of filename. Hugging Face caps directories at 10,000 files, and flat audio would blow past that. ## A note on the viewer The viewer only shows 110 rows because embedding audio bytes for all 1.15M rows would need roughly 340 GB of parquet, which is not a thing anyone wants to download just to browse. The full audio lives as separate WAVs in `audio/` and gets resolved when you `cast_column("audio", Audio())` after loading. ## License MIT.