Bangla Information Retrieval Test Collection

There are several IR test collections available in English (e.g. http://ir.dcs.gla.ac.uk/resources/test_collections/). Unfortunately, there is no Gold standard dataset available to test the effectiveness of Bangla IR. So, we have created a document collection containing 182 short stories, novels, and essays written by Rabindranath Tagore11 and 1000 newspaper articles published in 2013 crawled from the Bangla newspaper Prothom Alo12. The collection contains 100 newspaper articles each from one of the ten categories: বাংলাদেশ/ Bānlādēśa(EN: `Bangladesh'), খেলা/ khēlā(EN: `sports'), বিজ্ঞান ও প্রযুক্তি/ bijñāna ō prayukti(EN: `technology'), বিনোদন/ binōdana(EN: `entertainment'), আন্তর্জাতিক/ āntarjātika(EN: `international'), অর্থনীতি/ arthanīti(EN: `economy'), জীবনযাপন/ jībanayāpana(EN: `life-style'), মতামত/  matāmata(EN: `opinion'), শিক্ষা/ śikṣā(EN: `education') and আমরা/ āmarā(EN:`we-are'). There are 94 queries in the dataset, 26 queries belonging to complexity levels 1 and 2, 19 queries in complexity level 3 and 23 queries in complexity level 4. The definition of the complexity level of a query is described below:

Complexity Level 1: The query contains exact words, phrases or sentence from the document.

Complexity Level 2: The query is not present as it is in the document. There is a slight deviation.

Complexity Level 3: The query is a generalised phrase capturing the overall story or the document’s theme.

Complexity Level 4: It is a general query not related to any specific document.