ENTRP-SRCH: An Enterprise Search LTR-formatted dataset for ranking of search results

Fifteen judges from within an organisation (a University) have completed annotations for a total of 2544 Q-D pairs, covering the 20 most frequent topics (i.e. clusters of queries), and a formatted dataset is created. The LETOR format (aka LTR format) stipulates a relevance judgement in the first column, a query identifier (qid) in the second column, followed by a numbered feature vector array. The University’s new dataset is named ENTRP-SRCH, and an extract is shown below. Features 1 to 8 refer to BM25, recency, isAbout, isContact, view count, URLlength, LinkRank, and CTR, respectively. The consensus of the Q-D pair annotations was validated using an Inter Annotator Agreement study.