FRUCE V2

FRUCE-V2 is an extended version of FRUCE (Finding Relevant User Comments Evaluation Collection), which is an evaluation collection for IR systems created by the Semantic Access and Retrieval (SAR) group of ISL of FORTH. It is specifically designed for evaluating the retrieval of user reviews/comments about hotels. It contains 6 manually crafted queries, i.e. q1,...,q6 related to the topics of noise, cleanliness, and politeness and 40 manually annotated reviews (both manually designed and real ones from booking.com) for each query. The ratio of relevant/irrelevant (ci,qi) pairs is 1/3 for each query, constituting a collection of 240 such pairs in total.