Wembedder Wikidata-20170613-Truthy-Beta-Cbow-Size=100-Window=1-Min_Count=20

Wikidata embedding<br> ================== Gensim model:<br> wikidata-20170613-truthy-BETA-cbow-size=100-window=1-min_count=20 Download of Wikidata from:: https://dumps.wikimedia.org/wikidatawiki/entities/ Trigram construction:: from bz2 import BZ2File<br> import re dump_filename = 'wikidata-20170613-truthy-BETA.nt.bz2'<br> trigram_filename = 'wikidata-20170613-truthy-BETA.trigrams' pattern = re.compile(<br> (r'^&lt;http://www.wikidata.org/entity/(Q\d+)&gt; '<br> r'&lt;http://www.wikidata.org/prop/direct/(P\d+)&gt; '<br> r'&lt;http://www.wikidata.org/entity/(Q\d+)&gt;'),<br> flags=re.UNICODE) with open(trigram_filename, 'w') as f:<br> for line in BZ2File(dump_filename):<br> line = line.decode('utf-8')<br> match = pattern.search(line)<br> if match:<br> f.write(" ".join(match.groups()) + '\n') <br> Construction of Gensim model::<br> <br> import logging<br> from gensim.models import Word2Vec<br> from gensim.models.word2vec import LineSentence logging.basicConfig(<br> format='%(asctime)s : %(levelname)s : %(message)s',<br> level=logging.INFO) sentences = LineSentence('wikidata-20170613-truthy-BETA.trigrams') filename = 'wikidata-20170613-truthy-BETA-cbow-size=100-window=1-min_count=20'<br> w2v = Word2Vec(sentences, size=100, window=1, min_count=20, workers=10)<br> w2v.save(filename)