Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language

Turkronicles This repository presents a compilation of the publications of the Official Gazette of Türkiye (Resmi Gazete) and the records of the Grand National Assembly of Türkiye, from 1921 to 2024. Researchers can trace linguistic variation, semantic change, and stylistic trends from 1921 to 2024 in Turkish using the following resources: - Static embeddings- Contextual embeddings- N‑gram frequency lists- Raw text corpora Each resource is stored in a dedicated top‑level directory: /static_embeddings /contextual_embeddings/ngrams /corpus STATIC EMBEDDINGS The available static embeddings are as follows:-CBOW-PPMI -SVD Note: PPMI and SVD embeddings are stored as NumPy matrices whose rows correspond to word indices. Use the files under /static_embeddings/Vocabulary to map indices back to surface forms. In addition, CBOW embeddings are gensim Word2Vec models. CONTEXTUAL EMBEDDINGS Each decade checkpoint under /contextual_embeddings is a Hugging Face DistilBERT model fine‑tuned on that decade’s corpus. N‑GRAM FREQUENCY LISTS n-grams (1-,2-,3-,4-, and 5-gram) are provided for each decade. Each row in the files ending with .ngram is in the following format:<n‑gram> <frequency_count> CORPUS The /corpus/diachronic_mix directory contains decade-separated text files in UTF-8 encoding. The /corpus/sentences_fixed folder holds post-OCR-corrected, sentence-segmented files, with one sentence per line. Note that those who want to use the post-OCR-corrected text data should give extra attention, as the OCR correction method employed in this study imposes certain limitations (see the Limitations section for details). FILE METADATA The file metadata are stored in the /db_meta directory. OGT.csv holds metadata for the Official Gazette of Türkiye, and RGNAT.csv holds metadata for the parliamentary records. With these metadata files, you can easily distinguish between the two text types. If you use any part of this repository in academic work, please cite: @misc{yazar2024turkroniclesdiachronicresourcesfast, title={Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language}, author={Togay Yazar and Mucahid Kutlu and İsa Kerem Bayırlı}, year={2024}, eprint={2405.10133}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2405.10133}, }