wikipedia-2023-11-embed-multilingual-v3-int8-binary

# Multilingual Embeddings for Wikipedia in 300+ Languages (int8 & binary embeddings) This dataset contains the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset dump from 2023-11-01 from Wikipedia in all 300+ languages. The embeddings are provided as **int8** and **ubinary** that allow quick search and reduction of your vector index size up to 32. For more details, see [Cohere int8 & binary Embeddings](https://txt.cohere.com/int8-binary-embeddings/)  The individual articles have been chunked and embedded with the state-of-the-art multilingual [Cohere Embed V3 embedding model](https://txt.cohere.com/introducing-embed-v3/). This enables an easy way to semantically search across all of Wikipedia or to use it as a knowledge source for your RAG application. In total is it close to 250M paragraphs / embeddings. You can also use the model to perform cross-lingual search: Enter your search query in any language and get the most relevant results back. ## Loading the dataset ### Loading the document embeddings You can either load the dataset like this: ```python from datasets import load_dataset lang = "simple" #Use the Simple English Wikipedia subset docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binary", lang, split="train") ``` Or you can also stream it without downloading it before: ```python from datasets import load_dataset lang = "simple" #Use the Simple English Wikipedia subset docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binary", lang, split="train", streaming=True) for doc in docs: doc_id = doc['_id'] title = doc['title'] text = doc['text'] emb_int8 = doc['emb_int8'] #Int8 embeddings emb_ubinary = doc['emb_ubinary'] #(unsigned) binary embeddings ``` Note, depending on the language, the download can be quite large. ## Search - int8 embeddings int8 embeddings keep 99.7-100% of the search quality, while reducing your vector index 4x and speeding-up vector search up to 30%. Below example using [usearch](https://github.com/unum-cloud/usearch) to search on int8 embeddings. ```python from datasets import load_dataset import numpy as np import cohere from usearch.index import Index co = cohere.Client("<<YOUR_API_KEY>>") # Add your cohere API key from www.cohere.com lang = "simple" #Load at max 1000 chunks + embeddings max_docs = 1000 docs_stream = load_dataset(f"CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binary", lang, split="train", streaming=True) docs = [] doc_embeddings = [] for doc in docs_stream: docs.append(doc) doc_embeddings.append(doc['emb_int8']) if len(docs) >= max_docs: break doc_embeddings = np.asarray(doc_embeddings, dtype='int8') print(doc_embeddings.shape, doc_embeddings.dtype) #Create the usearch HNSW index on the int8 embeddings num_dim = 1024 index = Index(ndim=1024, metric='ip', dtype='i8') index.add(np.arange(len(doc_embeddings)), doc_embeddings) #Searching query = 'Who was Alan Turing' query_emb = co.embed(texts=[query], model="embed-multilingual-v3.0", input_type="search_query", embedding_types=["int8"]).embeddings query_emb = np.asarray(query_emb.int8, dtype='int8') #Search on the index and get top-10 results matches = index.search(query_emb, 10) # Print results for match in matches: doc_id = match.key print(docs[doc_id]['title']) print(docs[doc_id]['text']) print("----") ``` ## Search - Binary embeddings Binary embeddings keep 95% of the search quality, while reducing your vector index 32x and speeding-up vector search up to 40x. Below example using [faiss](https://github.com/facebookresearch/faiss) to search on binary embeddings. ```python #Run: pip install cohere datasets numpy faiss-cpu from datasets import load_dataset import numpy as np import cohere import faiss co = cohere.Client("<<YOUR_API_KEY>>") # Add your cohere API key from www.cohere.com lang = "simple" #Load at max 1000 chunks + embeddings max_docs = 1000 docs_stream = load_dataset(f"CohereLabs/wikipedia-2023-11-embed-multilingual-v3-int8-binary", lang, split="train", streaming=True) docs = [] doc_embeddings = [] for doc in docs_stream: docs.append(doc) doc_embeddings.append(doc['emb_ubinary']) if len(docs) >= max_docs: break doc_embeddings = np.asarray(doc_embeddings, dtype='uint8') #Create the faiss IndexBinaryFlat index num_dim = 1024 index = faiss.IndexBinaryFlat(num_dim) index.add(doc_embeddings) #Search def search(index, query, top_k=10): # Make sure to set input_type="search_query" query_emb = co.embed(texts=[query], model="embed-multilingual-v3.0", input_type="search_query", embedding_types=["ubinary", "float"]).embeddings query_emb_bin = np.asarray(query_emb.ubinary, dtype='uint8') query_emb_float = np.asarray(query_emb.float, dtype="float32") # Phase I: Search on the index with a binary hits_scores, hits_doc_ids = index.search(query_emb_bin, k=min(10*top_k, index.ntotal)) #Get the results in a list of hits hits = [{'doc_id': doc_id.item(), 'score_bin': score_bin} for doc_id, score_bin in zip(hits_doc_ids[0], hits_scores[0])] # Phase II: Do a re-scoring with the float query embedding binary_doc_emb = np.asarray([index.reconstruct(hit['doc_id']) for hit in hits]) binary_doc_emb_unpacked = np.unpackbits(binary_doc_emb, axis=-1).astype("int") binary_doc_emb_unpacked = 2*binary_doc_emb_unpacked-1 scores_cont = (query_emb_float[0] @ binary_doc_emb_unpacked.T) for idx in range(len(scores_cont)): hits[idx]['score_cont'] = scores_cont[idx] #Sort by largest score_cont hits.sort(key=lambda x: x['score_cont'], reverse=True) return hits[0:top_k] query = 'Who was Alan Turing' hits = search(index, query) # Print results print("Query:", query) for hit in hits: doc_id = hit['doc_id'] print(docs[doc_id]['title']) print(docs[doc_id]['text']) print(docs[doc_id]['url'], "\n") ``` ## Overview The following table contains all language codes together with the total numbers of passages. | Language | #Docs | |---|:---:| | en | 41,488,110 | | de | 20,772,081 | | fr | 17,813,768 | | ru | 13,734,543 | | es | 12,905,284 | | it | 10,462,162 | | ceb | 9,818,657 | | uk | 6,901,192 | | ja | 6,626,537 | | nl | 6,101,353 | | pl | 5,973,650 | | pt | 5,637,930 | | sv | 4,911,480 | | ca | 4,156,889 | | ar | 3,691,451 | | cs | 3,118,524 | | he | 2,948,882 | | hu | 2,924,609 | | vi | 2,835,049 | | zh | 2,775,260 | | fi | 2,427,097 | | id | 2,358,335 | | no | 2,211,270 | | sr | 2,154,631 | | fa | 2,073,154 | | tr | 1,775,036 | | ro | 1,770,527 | | el | 1,599,770 | | ko | 1,513,291 | | bg | 1,455,765 | | hy | 1,386,140 | | eu | 1,327,579 | | da | 1,224,982 | | eo | 1,216,706 | | war | 1,185,097 | | sh | 1,139,309 | | tt | 1,119,546 | | arz | 1,089,164 | | gl | 1,056,990 | | et | 1,054,770 | | ce | 1,013,217 | | ast | 1,010,445 | | sl | 984,855 | | hr | 910,923 | | sk | 874,014 | | ms | 869,579 | | be | 857,033 | | th | 839,712 | | az | 817,631 | | uz | 811,028 | | mk | 784,576 | | lt | 770,069 | | bn | 767,965 | | cy | 762,338 | | ta | 685,975 | | simple | 646,424 | | te | 634,778 | | kk | 627,085 | | ka | 595,401 | | hi | 541,822 | | nn | 530,590 | | lv | 484,957 | | af | 461,674 | | ba | 434,939 | | ur | 434,269 | | bs | 396,692 | | sq | 388,788 | | ml | 384,795 | | min | 373,156 | | la | 340,521 | | pnb | 335,958 | | be-x-old | 314,600 | | kn | 309,085 | | azb | 294,248 | | oc | 283,297 | | zh-min-nan | 278,547 | | fy | 248,075 | | my | 241,025 | | lb | 216,558 | | ky | 216,344 | | als | 206,387 | | mr | 203,479 | | br | 200,486 | | pa | 188,091 | | is | 177,272 | | mg | 171,947 | | sw | 171,650 | | ha | 167,807 | | tl | 166,907 | | nds | 166,019 | | an | 143,163 | | jv | 142,104 | | ps | 138,240 | | ig | 132,250 | | new | 128,696 | | tg | 128,237 | | ga | 125,456 | | lld | 125,094 | | su | 124,390 | | cv | 122,671 | | ckb | 120,886 | | si | 119,223 | | mn | 114,878 | | lmo | 103,836 | | io | 101,713 | | gu | 99,450 | | vec | 95,072 | | zh-yue | 89,145 | | bar | 88,238 | | sco | 83,906 | | ne | 83,598 | | ku | 82,935 | | hyw | 82,343 | | pms | 77,834 | | as | 76,093 | | km | 74,177 | | sah | 71,599 | | li | 69,267 | | or | 65,510 | | mt | 64,038 | | szl | 56,836 | | yi | 55,375 | | ht | 55,079 | | dag | 53,343 | | sa | 51,735 | | nv | 49,355 | | bpy | 47,757 | | vo | 47,375 | | ug | 44,764 | | sat | 43,500 | | ia | 42,012 | | bo | 41,438 | | mwl | 41,273 | | sd | 40,395 | | bcl | 39,967 | | mnw | 39,578 | | hsb | 39,560 | | avk | 39,001 | | scn | 38,359 | | rm | 37,436 | | diq | 34,743 | | vep | 33,654 | | xmf | 33,238 | | ban | 32,320 | | wa | 32,132 | | ilo | 31,046 | | nds-nl | 30,918 | | qu | 30,529 | | so | 29,936 | | mhr | 29,619 | | vls | 29,227 | | sc | 28,977 | | fo | 28,809 | | gd | 28,149 | | rw | 28,037 | | gom | 27,792 | | yo | 27,789 | | tum | 26,743 | | wuu | 26,532 | | frr | 26,010 | | sn | 25,941 | | tk | 24,269 | | blk | 24,194 | | mzn | 23,837 | | co | 23,065 | | szy | 22,854 | | am | 22,467 | | shn | 22,432 | | skr | 21,081 | | lfn | 20,781 | | tyv | 20,762 | | lij | 20,553 | | ie | 19,994 | | rue | 19,916 | | crh | 19,016 | | gor | 18,146 | | ary | 17,463 | | dv | 16,941 | | lg | 16,751 | | roa-tara | 16,572 | | bjn | 16,429 | | tw | 16,304 | | bh | 15,938 | | pam | 15,134 | | os | 15,096 | | myv | 15,062 | | gn | 14,983 | | lez | 14,152 | | mai | 13,806 | | kv | 13,534 | | pcd | 13,057 | | zh-classical | 12,791 | | zea | 12,528 | | lo | 12,525 | | gv | 12,074 | | stq | 11,890 | | zu | 11,680 | | smn | 11,672 | | kw | 11,539 | | bat-smg | 11,240 | | hif | 11,215 | | ext | 10,967 | | ace | 10,821 | | trv | 10,546 | | ami | 10,538 | | tcy | 10,531 | | lad | 10,386 | | alt | 10,256 | | pap | 10,187 | | kab | 10,179 | | fur | 10,148 | | nap | 10,079 | | mrj | 9,771 | | kaa | 9,548 | | nqo | 9,153 | | glk | 9,120 | | pfl | 8,790 | | fiu-vro | 8,757 | | nso | 8,635 | | jbo | 8,577 | | bxr | 8,549 | | wo | 8,549 | | olo | 8,530 | | map-bms | 8,393 | | ksh | 8,226 | | csb | 8,085 | | av | 7,873 | | mni | 7,740 | | udm | 7,730 | | mi | 7,643 | | kbp | 7,616 | | dsb | 7,536 | | frp | 7,294 | | om | 7,045 | | ang | 7,023 | | hak | 6,866 | | gur | 6,761 | | se | 6,733 | | anp | 6,704 | | tay | 6,434 | | mdf | 6,351 | | gcr | 6,347 | | koi | 6,300 | | krc | 6,293 | | ay | 5,985 | | cdo | 5,917 | | nrm | 5,786 | | xh | 5,756 | | tn | 5,712 | | tly | 5,598 | | shi | 5,179 | | pcm | 5,076 | | fat | 4,968 | | nia | 4,795 | | dty | 4,728 | | kbd | 4,667 | | gpe | 4,289 | | cbk-zam | 4,224 | | ff | 4,166 | | dz | 4,117 | | guw | 3,982 | | eml | 3,979 | | ln | 3,774 | | inh | 3,768 | | nah | 3,720 | | ab | 3,465 | | ks | 3,255 | | mad | 3,236 | | haw | 3,227 | | gag | 3,076 | | tet | 3,030 | | ny | 2,933 | | pag | 2,727 | | guc | 2,454 | | roa-rup | 2,409 | | jam | 2,387 | | awa | 2,242 | | pdc | 2,239 | | to | 2,165 | | za | 2,132 | | st | 2,051 | | ltg | 2,005 | | atj | 1,967 | | nov | 1,916 | | ss | 1,904 | | pwn | 1,881 | | ee | 1,819 | | sm | 1,659 | | ts | 1,645 | | gan | 1,626 | | xal | 1,619 | | kcg | 1,555 | | cu | 1,477 | | srn | 1,395 | | got | 1,280 | | fon | 1,247 | | din | 1,214 | | arc | 1,167 | | fj | 1,164 | | rmy | 1,113 | | ady | 1,040 | | rn | 1,033 | | bm | 1,017 | | tpi | 957 | | ve | 919 | | ki | 798 | | pnt | 796 | | chr | 788 | | kl | 770 | | lbe | 766 | | bi | 718 | | ti | 706 | | kg | 609 | | pih | 606 | | ch | 513 | | bug | 429 |emb | ty | 297 | | ik | 275 | | iu | 263 | | pi | 260 | | sg | 204 | | chy | 57 | | cr | 41 | | Total | 247,154,006 |