Online Newspaper Data Retrieval and Cleaning

This project focused on systematically extracting and refining a comprehensive collection of newspaper articles on the Indian River Linking Project (INRLP) from major Indian English-language newspapers indexed in the LexisNexis API between 2004 and 2021. The INRLP seeks to transfer water from so-called “surplus” areas that are flood-prone to “deficit” areas that are otherwise drought-prone. A similar protocol and code structure can be used to retrieve and clean digital newspapers available in the LexisNexis API for any disaster context. The structure of our protocol can be broken down into 6 stages, which can be found in the data retrieval and data cleaning writeups:
Stage 1: Choosing Keywords 
Stage 2: Authentication, Search, and Retrieval of Results 
Stage 3: Collecting and Parsing Results
Stage 4: Removing duplicates that are exact character matches 
Stage 5: Grouping duplicates with similar content 
Stage 6: Removing duplicates with similar content 
This project aims to provide a foundation for retrieving and analyzing several decades' worth of media coverage through a combination of automated and manual methods. As news and other textual outlets are increasingly being "born digital," there are opportunities for media analyses that break barriers to data accessibility and bolster interdisciplinary research.