CoST: An annotated Data Collection for Complex Search

CoST is a novel richly annotated dataset for evaluating complex search tasks, collaboratively designed by researchers from the computer science and cognitive psychology domains, and intended to answer a wide range of research questions dealing with task-based search. CoST includes 5667 queries recorded in 630 task-based sessions that result from a user study involving 70 french native participants who are expert in one among 3 different domains of expertise (computer science, medicine, psychology). Each participant completed 15 tasks with 5 different types of cognitive complexity (fact-finding, exploratory learning, decisionmaking, problem-solving, multicriteria-inferential). In addition to search data (e.g., queries and clicks), CoST provides task and sessionrelated data, task annotations and query annotations. We illustrate possible usages of CoST through the evaluation of query classification models and the understanding of the effect of task complexity and domain on user’s search behavior.<br>If you use this dataset, please cite:<br>@inproceedings{10.1145/3459637.3481998,author = {Dosso, Cheyenne and Moreno, Jose G. and Chevalier, Aline and Tamine, Lynda},title = {CoST: An Annotated Data Collection for Complex Search},year = {2021},isbn = {9781450384469},publisher = {Association for Computing Machinery},address = {New York, NY, USA},url = {https://doi.org/10.1145/3459637.3481998},doi = {10.1145/3459637.3481998},booktitle = {Proceedings of the 30th ACM International Conference on Information &amp; Knowledge Management},pages = {4455–4464},numpages = {10},keywords = {complex search task, user study, evaluation, expertise},location = {Virtual Event, Queensland, Australia},series = {CIKM '21}}<br>