sliderforthewin/lt-asr-lm-corpora

---
language:
- lt
license: cc0-1.0
pretty_name: Lithuanian ASR LM Training Corpora
tags:
- language-model
- lithuanian
- asr
---

# Lithuanian ASR Language Model Training Corpora

Text corpora for building token-level n-gram language models for
Lithuanian ASR beam search decoding. Used with
[sliderforthewin/parakeet-tdt-lt](https://huggingface.co/sliderforthewin/parakeet-tdt-lt).

## Corpora

| file | sentences | source | register |
|------|-----------|--------|----------|
| cc100.sentences.txt.zst | 56.9M | CC-100 web crawl | noisy web |
| wikipedia.sentences.txt.zst | 2.9M | Lithuanian Wikipedia | encyclopedic |
| opensubtitles.sentences.txt.zst | 1.3M | OpenSubtitles | informal dialogue |
| europarl.sentences.txt.zst | 634k | EU Parliament (Europarl) | formal parliamentary |

## Best LM recipe

The **domain 5-gram** (Wikipedia + training manifests only) achieves
9.51% WER on CV25 LT — better than using all 61M sentences. Domain
match matters more than corpus size.

Adding Europarl helps VoxPopuli (parliamentary speech benchmark).

See in the finetuneparakeet repo for the full source
guide and build instructions.

## Usage

--- Building KenLM tools (one-time setup) ---
-- Looking for clock_gettime in c
-- Looking for clock_gettime in c - found
-- Configuring done (4.4s)
-- Generating done (0.2s)
-- Build files have been written to: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build
[ 98%] Linking CXX executable ../../bin/lmplz
[ 98%] Built target lmplz
12 warnings generated.
[100%] Linking CXX executable ../../bin/count_ngrams
[100%] Built target count_ngrams
Built: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build/bin/lmplz
=== Build parakeet token-level 5-gram LM from HF corpora ===
repo: sliderforthewin/lt-asr-lm-corpora
output: data/lm/lt_token_5gram_v2.arpa
lmplz: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build/bin/lmplz
lmplz mem: 50%
prune: 0 2 2 2 2

--- Step 1: Downloading corpora from HuggingFace ---
/Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/data/lm/hf-corpora
Downloaded to data/lm/hf-corpora/corpora/

--- Step 2: Decompressing and concatenating corpora ---
Combining into data/lm/lt_combined_hf.txt...
+cc100.sentences.txt.zst... 56892247 lines
+opensubtitles.sentences.txt.zst... 1269060 lines
+wikipedia.sentences.txt.zst... 2938541 lines
Total: 61099848 lines, 10801 MB

--- Step 3: Stream-tokenize + build 5-gram LM via lmplz ---
Piping 61099848 sentences through parakeet BPE into lmplz.
Memory: ~4 GB (tokenizer) + 50% (lmplz sort buffers)