wikipedia-2023-11-embed-multilingual-v3

# Multilingual Embeddings for Wikipedia in 300+ Languages This dataset contains the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset dump from 2023-11-01 from Wikipedia in all 300+ languages. The individual articles have been chunked and embedded with the state-of-the-art multilingual [Cohere Embed V3 embedding model](https://txt.cohere.com/introducing-embed-v3/). This enables an easy way to semantically search across all of Wikipedia or to use it as a knowledge source for your RAG application. In total is it close to 250M paragraphs / embeddings. You can also use the model to perform cross-lingual search: Enter your search query in any language and get the most relevant results back. ## Loading the dataset ### Loading the document embeddings You can either load the dataset like this: ```python from datasets import load_dataset lang = "simple" #Use the Simple English Wikipedia subset docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train") ``` Or you can also stream it without downloading it before: ```python from datasets import load_dataset lang = "simple" #Use the Simple English Wikipedia subset docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train", streaming=True) for doc in docs: doc_id = doc['_id'] title = doc['title'] text = doc['text'] emb = doc['emb'] ``` Note, depending on the language, the download can be quite large. ## Search A full search example (on the first 1,000 paragraphs): ```python #Run: pip install cohere datasets numpy from datasets import load_dataset import numpy as np import cohere co = cohere.Client("YOUR_COHERE_API_KEY") # Add your cohere API key from www.cohere.com lang = "simple" top_k = 3 #Load at max 1000 chunks + embeddings max_docs = 1000 docs_stream = load_dataset(f"CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train", streaming=True) docs = [] doc_embeddings = [] for doc in docs_stream: docs.append(doc) doc_embeddings.append(doc['emb']) if len(docs) >= max_docs: break doc_embeddings = np.asarray(doc_embeddings) query = 'Who was Alan Turing' response = co.embed(texts=[query], model='embed-multilingual-v3.0', input_type="search_query") query_embedding = response.embeddings query_embedding = np.asarray(query_embedding) # Compute dot score between query embedding and document embeddings dot_scores = np.matmul(query_embedding, doc_embeddings.transpose())[0] top_k_hits = np.argpartition(dot_scores, -top_k)[-top_k:].tolist() # Sort top_k_hits by dot score top_k_hits.sort(key=lambda x: dot_scores[x], reverse=True) # Print results print("Query:", query) for doc_id in top_k_hits: print(docs[doc_id]['title']) print(docs[doc_id]['text']) print(docs[doc_id]['url'], "\n") ``` ## Overview The following table contains all language codes together with the total numbers of passages. | Language | #Docs | |---|:---:| | en | 41,488,110 | | de | 20,772,081 | | fr | 17,813,768 | | ru | 13,734,543 | | es | 12,905,284 | | it | 10,462,162 | | ceb | 9,818,657 | | uk | 6,901,192 | | ja | 6,626,537 | | nl | 6,101,353 | | pl | 5,973,650 | | pt | 5,637,930 | | sv | 4,911,480 | | ca | 4,156,889 | | ar | 3,691,451 | | cs | 3,118,524 | | he | 2,948,882 | | hu | 2,924,609 | | vi | 2,835,049 | | zh | 2,775,260 | | fi | 2,427,097 | | id | 2,358,335 | | no | 2,211,270 | | sr | 2,154,631 | | fa | 2,073,154 | | tr | 1,775,036 | | ro | 1,770,527 | | el | 1,599,770 | | ko | 1,513,291 | | bg | 1,455,765 | | hy | 1,386,140 | | eu | 1,327,579 | | da | 1,224,982 | | eo | 1,216,706 | | war | 1,185,097 | | sh | 1,139,309 | | tt | 1,119,546 | | arz | 1,089,164 | | gl | 1,056,990 | | et | 1,054,770 | | ce | 1,013,217 | | ast | 1,010,445 | | sl | 984,855 | | hr | 910,923 | | sk | 874,014 | | ms | 869,579 | | be | 857,033 | | th | 839,712 | | az | 817,631 | | uz | 811,028 | | mk | 784,576 | | lt | 770,069 | | bn | 767,965 | | cy | 762,338 | | ta | 685,975 | | simple | 646,424 | | te | 634,778 | | kk | 627,085 | | ka | 595,401 | | hi | 541,822 | | nn | 530,590 | | lv | 484,957 | | af | 461,674 | | ba | 434,939 | | ur | 434,269 | | bs | 396,692 | | sq | 388,788 | | ml | 384,795 | | min | 373,156 | | la | 340,521 | | pnb | 335,958 | | be-x-old | 314,600 | | kn | 309,085 | | azb | 294,248 | | oc | 283,297 | | zh-min-nan | 278,547 | | fy | 248,075 | | my | 241,025 | | lb | 216,558 | | ky | 216,344 | | als | 206,387 | | mr | 203,479 | | br | 200,486 | | pa | 188,091 | | is | 177,272 | | mg | 171,947 | | sw | 171,650 | | ha | 167,807 | | tl | 166,907 | | nds | 166,019 | | an | 143,163 | | jv | 142,104 | | ps | 138,240 | | ig | 132,250 | | new | 128,696 | | tg | 128,237 | | ga | 125,456 | | lld | 125,094 | | su | 124,390 | | cv | 122,671 | | ckb | 120,886 | | si | 119,223 | | mn | 114,878 | | lmo | 103,836 | | io | 101,713 | | gu | 99,450 | | vec | 95,072 | | zh-yue | 89,145 | | bar | 88,238 | | sco | 83,906 | | ne | 83,598 | | ku | 82,935 | | hyw | 82,343 | | pms | 77,834 | | as | 76,093 | | km | 74,177 | | sah | 71,599 | | li | 69,267 | | or | 65,510 | | mt | 64,038 | | szl | 56,836 | | yi | 55,375 | | ht | 55,079 | | dag | 53,343 | | sa | 51,735 | | nv | 49,355 | | bpy | 47,757 | | vo | 47,375 | | ug | 44,764 | | sat | 43,500 | | ia | 42,012 | | bo | 41,438 | | mwl | 41,273 | | sd | 40,395 | | bcl | 39,967 | | mnw | 39,578 | | hsb | 39,560 | | avk | 39,001 | | scn | 38,359 | | rm | 37,436 | | diq | 34,743 | | vep | 33,654 | | xmf | 33,238 | | ban | 32,320 | | wa | 32,132 | | ilo | 31,046 | | nds-nl | 30,918 | | qu | 30,529 | | so | 29,936 | | mhr | 29,619 | | vls | 29,227 | | sc | 28,977 | | fo | 28,809 | | gd | 28,149 | | rw | 28,037 | | gom | 27,792 | | yo | 27,789 | | tum | 26,743 | | wuu | 26,532 | | frr | 26,010 | | sn | 25,941 | | tk | 24,269 | | blk | 24,194 | | mzn | 23,837 | | co | 23,065 | | szy | 22,854 | | am | 22,467 | | shn | 22,432 | | skr | 21,081 | | lfn | 20,781 | | tyv | 20,762 | | lij | 20,553 | | ie | 19,994 | | rue | 19,916 | | crh | 19,016 | | gor | 18,146 | | ary | 17,463 | | dv | 16,941 | | lg | 16,751 | | roa-tara | 16,572 | | bjn | 16,429 | | tw | 16,304 | | bh | 15,938 | | pam | 15,134 | | os | 15,096 | | myv | 15,062 | | gn | 14,983 | | lez | 14,152 | | mai | 13,806 | | kv | 13,534 | | pcd | 13,057 | | zh-classical | 12,791 | | zea | 12,528 | | lo | 12,525 | | gv | 12,074 | | stq | 11,890 | | zu | 11,680 | | smn | 11,672 | | kw | 11,539 | | bat-smg | 11,240 | | hif | 11,215 | | ext | 10,967 | | ace | 10,821 | | trv | 10,546 | | ami | 10,538 | | tcy | 10,531 | | lad | 10,386 | | alt | 10,256 | | pap | 10,187 | | kab | 10,179 | | fur | 10,148 | | nap | 10,079 | | mrj | 9,771 | | kaa | 9,548 | | nqo | 9,153 | | glk | 9,120 | | pfl | 8,790 | | fiu-vro | 8,757 | | nso | 8,635 | | jbo | 8,577 | | bxr | 8,549 | | wo | 8,549 | | olo | 8,530 | | map-bms | 8,393 | | ksh | 8,226 | | csb | 8,085 | | av | 7,873 | | mni | 7,740 | | udm | 7,730 | | mi | 7,643 | | kbp | 7,616 | | dsb | 7,536 | | frp | 7,294 | | om | 7,045 | | ang | 7,023 | | hak | 6,866 | | gur | 6,761 | | se | 6,733 | | anp | 6,704 | | tay | 6,434 | | mdf | 6,351 | | gcr | 6,347 | | koi | 6,300 | | krc | 6,293 | | ay | 5,985 | | cdo | 5,917 | | nrm | 5,786 | | xh | 5,756 | | tn | 5,712 | | tly | 5,598 | | shi | 5,179 | | pcm | 5,076 | | fat | 4,968 | | nia | 4,795 | | dty | 4,728 | | kbd | 4,667 | | gpe | 4,289 | | cbk-zam | 4,224 | | ff | 4,166 | | dz | 4,117 | | guw | 3,982 | | eml | 3,979 | | ln | 3,774 | | inh | 3,768 | | nah | 3,720 | | ab | 3,465 | | ks | 3,255 | | mad | 3,236 | | haw | 3,227 | | gag | 3,076 | | tet | 3,030 | | ny | 2,933 | | pag | 2,727 | | guc | 2,454 | | roa-rup | 2,409 | | jam | 2,387 | | awa | 2,242 | | pdc | 2,239 | | to | 2,165 | | za | 2,132 | | st | 2,051 | | ltg | 2,005 | | atj | 1,967 | | nov | 1,916 | | ss | 1,904 | | pwn | 1,881 | | ee | 1,819 | | sm | 1,659 | | ts | 1,645 | | gan | 1,626 | | xal | 1,619 | | kcg | 1,555 | | cu | 1,477 | | srn | 1,395 | | got | 1,280 | | fon | 1,247 | | din | 1,214 | | arc | 1,167 | | fj | 1,164 | | rmy | 1,113 | | ady | 1,040 | | rn | 1,033 | | bm | 1,017 | | tpi | 957 | | ve | 919 | | ki | 798 | | pnt | 796 | | chr | 788 | | kl | 770 | | lbe | 766 | | bi | 718 | | ti | 706 | | kg | 609 | | pih | 606 | | ch | 513 | | bug | 429 | | ty | 297 | | ik | 275 | | iu | 263 | | pi | 260 | | sg | 204 | | chy | 57 | | cr | 41 | | Total | 247,154,006 |