Twitter pre-trained word vectors

Clean up of <strong>glove.twitter.27B.zip </strong>&lt;ODC Public Domain Dedication and Licence (PDDL) 1.0&gt;.

"2B tweets, 27B tokens, 1.2M vocab, uncased"

<strong>Changes from original</strong>

Headers added to allow loading by gensim. [Added via scripts.glove2word2vec]
Recompressed as individual gzip files [Instead of a combined zip].

These changes make the files easier to work with and increase compatibility.

<strong>Headers</strong>

Example of added header line

1193513 200

Header gives number of tokens and dimensions.

<strong>Statistics</strong>

Entries: 1,193,513
Token length (characters). Min:1, Max:140, Avg:6.73
Number of words per token. Min:0, Max:17, Avg:1.00669200921984
Tokens with more than one word: 4874 (0.41%)
Twitter data collection date: <em>Unknown.</em><strong><em> </em></strong>

History:

<strong>?? Aug 2014 </strong>— GloVe v.1.0 released
<strong>16 Aug 2014 </strong>— Files first appear as headerless .txt.gz files, some files have mislabeled linked (via wayback machine)
<strong>?? Oct 2015</strong> — GloVe v.1.2 released
<strong>?? ??? ????</strong> — Files replaced with a .zip file
<strong>03 June 2019 </strong>— (These files) Repackaged like original as .txt.gz, plus added headers for increased compatibility

Example of 17-word token:

سكس_طيز_قحبه_عنيف_اغتصاب_سكسيه_فحل_زب_نيك_بنات_مكوه_شهوه_لحس_عنف_تومبوي_ليدي_سبورت

<strong>200d file:</strong>

Normalized: no
Values: Min:-6.7986, Max:4.609, Avg:0.009065093
Zero values (exactly zero): none
Zero values (approx zero) per entry: Min:0 (0.00%), Max:2 of 200 (1.0%), Avg:0.00375865197949247 (0.00%)