RAG retrieves the quality assessment dataset

<b><i>This dataset aims to assess the impact of retrieval quality on the generation effect of the Retrieval Enhancement generation (RAG) system, and it contains a total of 728 samples. During the data collection process, we screened out 728 real user queries of different complexities from the public question-answering datasets (including SQuAD, Natural Questions and WebQuestions), covering various types such as factual questions and answers, multi-hop reasoning, and opinion queries. For each query, a hybrid retrieval strategy based on BM25 and Sentence-BERT is used to obtain relevant documents from the Wikipedia corpus, retrieving 5 to 15 documents for each query.</i></b><b><i>During the data processing stage, we extracted eight key features through a combination of manual annotation and automatic calculation: three annotators independently evaluated the query complexity (1 to 10 points) and took the average. Document relevance is achieved by fusing the relevance score based on annotations (0-1) with the predicted values of the pre-trained model. The semantic similarity between documents is calculated through Sentence-BERT, and the retrieval diversity features are generated by combining the diversity score based on TF-IDF. Automatically calculate the coverage of key entities using the entity linking tool; Quantify information redundancy based on the degree of overlap of document content; Record the actual number of retrieved documents as the retrieval depth; Finally, five experts will rate the generated responses from three dimensions: relevance, accuracy and completeness (0 to 100 points), which will serve as the final quality label. All eigenvalues have undergone standardized processing to ensure that the data range is reasonable and in line with the actual application scenarios, providing a real and reliable research basis for the optimization of the retrieval strategy and quality assessment of the RAG system.</i></b>