Universities online content corpus

Between April and October 2018, the Internet domains of 50 elite research universities in the US were scraped using the Scrapy framework for automated crawling and extraction of data from websites. The list of universities scraped was taken from the top 50 entries in the US News University Ranking Charts of 2017. The scraping process started at the base URL of each University domain and proceeded to extract textual elements contained in a URL endpoint and to follow all the detected links pointing within the University domain to continue collecting textual elements up to a predefined depth level. A depth first crawling algorithm for visiting scraped links was followed for memory efficiency reasons. In total, a text corpus of size 43 GB for the 50 universities combined was retrieved.