MS Marco Optimal Crawling Dataset

The dataset used for Optimal Freshness Crawl Under Politeness Constraints and Staying up to Date with Online Content Changes Using Reinforcement Learning for Scheduling which are both focused on providing an optimal crawling schedule for a search engine based ont he changing nature of the internet.

本数据集适用于两项研究，其标题分别为《基于礼貌性约束的最优新鲜度爬取》（Optimal Freshness Crawl Under Politeness Constraints）与《利用强化学习（Reinforcement Learning）进行调度以紧跟在线内容更新》（Staying up to Date with Online Content Changes Using Reinforcement Learning for Scheduling）；两项研究均旨在基于互联网的动态变化特性，为搜索引擎提供最优的爬取调度方案。