Mining Web Evolution: A Dataset of DOM Changes Across Websites

Abstract: Continuous evolution in web applications leads to frequent structural modifications, presenting significant challenges for software maintenance and empirical analysis. However, comprehensive datasets reflecting large-scale, real-world web evolution remain limited. This paper introduces a dataset comprising historical Document Object Model (DOM) snapshots from 226 widely accessed websites, retrieved via the Wayback Machine API. For each site, up to 1,000 time-ordered versions were collected at a minimum interval of one day, capturing detailed structural data such as absolute XPath positions and precise timestamps. The dataset encompasses over 251 million DOM node positions, providing a robust foundation for studies on change patterns, predictive modelling of web evolution, and benchmarking maintenance tools. By releasing this resource to the research community, we facilitate reproducible experimentation and enable novel, data-driven investigations into the long-term evolution of web applications. Dataset Description: The dataset is provided as CSV files using the separator character ','. Each file is named after the project or website name. We provide two directories, one for normalised values and one without normalised values. Each one of the files contains 11 column representing features of a node. node value timestamp position length depth nr_siblings nr_children xpath isattribute changed node is the name of the node. If it's a structural node, then it's a tag name; if an attribute, then attribute name. The value is used for attributes which have information within. timestamp is which snapshot the node was taken from the Wayback Machine. position is the relative position of the node within the neighbours. length is how far the node is from the root, and depth is how far it is from the closest leaf node. nr_siblings is the number of nodes within the same structural place. nr_children is the number of nodes within the current node. xpath is the path from the root to the node. isattribute is true or false to indicate whether the node is an attribute. Lastly, changed is the label given to the node; has it changed from the last version or not. We provide two zip files: DOM_position_change_normalised : contains the data for training and testing model-driven solutions. The values computed in the files are normalised using MinMaxScaler from sklearn. DOM_position_change : Contains the raw data mined from the DOMs of websites of the study. And a dataset_analysis.ipynb file which is a notebook that shows how the figures in the paper were processed from the data.