Relations Between Relevance Assessments, Bibliometrics and Altmetrics

This archive contains the accompanying data and source code of our study 'Relations Between Relevance Assessments, Bibliometrics and Altmetrics' submitted to BIR 2020 @ ECIR 2020. In order to reproduce our study, the iSearch collection is needed. After downloading this archive, place the folders `PF/` and `PN/` in the root directory. The directory `spreadsheets` contains the results of our final evaluation. The required data for these results can be found in the directory `data`, that contains the output of the scripts (in the directory `src`).

Abstract:

Relevance assessment in retrieval test collections and citations/mentions of scientific documents are two different forms of relevance decisions. To investigate the relations between these direct and indirect forms of relevance decisions, we combine arXiv data with Web of Science and Altmetrics data. In this new collection, we assess the effect of relevance ratings on measured perception in the form of citations or mentions, likes, tweets, et cetera. The impact of our work is that we could show a relation between direct relevance assessments and indirect relevance signals.