Data for "SuperSim: a test set for word similarity and relatedness in Swedish"

This repository contains the data described in <em>SuperSim: a test set for word similarity and relatedness in Swedish </em>(Hengchen and Tahmasebi, 2021) available at https://aclanthology.org/2021.nodalida-main.27/ . If you use part or whole of this resource, please cite the following work or alternatively use the bibtex entry: Hengchen, Simon and Tahmasebi, Nina, 2021. SuperSim: a test set for word similarity and relatedness in Swedish. In <em>The 23rd Nordic Conference on Computational Linguistics (NoDaLiDa’21)</em>. <pre><code>@inproceedings{hengchen-tahmasebi-2021-supersim, title = "{SuperSim:} a test set for word similarity and relatedness in {Swedish}", author = "Hengchen, Simon and Tahmasebi, Nina", booktitle = "Proceedings of the 23rd Nordic Conference on Computational Linguistics", month = may # "{--}" # jun, year = "2021", address = "Reykjavik, Iceland, and Online", publisher = {Link{\"o}ping University Electronic Press}, }</code></pre> The data contained in this repository is as follows: The <code>code</code> folder contains: <code>main.py</code> <code>utils.py</code> <code>train_base_models.py</code> <code>perl-clean.pl</code> <code>requirements.txt</code> The <code>data</code> folder contains: <code>gold_relatedness.tsv</code>: all relatedness judgments from all annotators, as well as the mean <code>gold_similarity.tsv</code>: all similarity judgments from all annotators, as well as the mean <code>models</code> contains baseline models: Trained on the Swedish Gigaword: FastText: <code>gigaword_sv.ft</code> (and <code>gigaword_sv.ft.trainables.syn1neg.npy</code>, <code>gigaword_sv.ft.trainables.vectors_ngrams_lockf.npy</code>, <code>gigaword_sv.ft.trainables.vectors_vocab_lockf.npy</code>, <code>gigaword_sv.ft.wv.vectors_ngrams.npy</code>, <code>gigaword_sv.ft.wv.vectors_vocab.npy</code>, <code>gigaword_sv.ft.wv.vectors.npy</code>) Word2Vec: <code>gigaword_sv.w2v</code> (and <code>gigaword_sv.w2v.trainables.syn1neg.npy</code>, <code>gigaword_sv.w2v.wv.vectors.npy</code>) GloVe: <code>glove_vectors_giga.txt</code> and <code>glove_vocab_giga.txt</code> Trained on Swedish Wikipedia: FastText: <code>wiki_sv.ft</code> (and <code>wiki_sv.ft.trainables.syn1neg.npy</code>, <code>wiki_sv.ft.trainables.vectors_ngrams_lockf.npy</code>, <code>wiki_sv.ft.trainables.vectors_vocab_lockf.npy</code>, <code>wiki_sv.ft.wv.vectors_ngrams.npy</code>, <code>wiki_sv.ft.wv.vectors.npy</code>, <code>wiki_sv.ft.wv.vectors_vocab.npy</code>) Word2Vec: <code>wiki_sv.w2v</code> (and <code>wiki_sv.w2v.trainables.syn1neg.npy</code>, <code>wiki_sv.w2v.wv.vectors.npy</code>) GloVe: <code>glove_vectors_WIKI.txt</code> and <code>glove_vocab_WIKI.txt</code> <code>corpora</code>: The Swedish Gigaword corpus can be downloaded, along with code, from: https://spraakbanken.gu.se/en/resources/gigaword. We created our corpus with <code>python extract_bw.py --mode plain outfile.txt</code>. <code>sv_wiki.gensim</code> is a cleaned Swedish Wikipedia dump from 2020/10/20 (originally <code>svwiki-20201020-pages-articles.xml</code>) and one of our baseline corpora. Details on annotation procedures are available in the paper. <strong>Acknowledgments</strong>: This work has been funded in part by the project <em>Towards Computational Lexical Semantic Change Detection</em> supported by the Swedish Research Council (2019--2022; dnr 2018-01184), and <em>Nationella Språkbanken</em> (the Swedish National Language Bank), jointly funded by the Swedish Research Council (2018--2024; dnr 2017-00626) and its ten partner institutions.