Albanian corpus used for pertaining large language models

The corpus used for pertaining large language models is an SQLite (v3) database with the following tables:

corp_src: the sources of the Albanian text

corp_doc: the corpus source (names) and source files

doc: joins from sentences to corpus document source (`corp_doc`)

sent: the Albanian sentences with tokenization and token length

This query shows how to get the corpus sources and constituent counts:

select cs.id as name, cs.url, count(*) as count

  from corp_src as cs, corp_doc as cd, doc as d, sent as s

  where cd.name = cs.id and

        cd.doc_id = d.rowid and

cd.doc_id = s.doc_id

  group by cs.id;