Development and Implementation of Database and Analyses for High Frequency Data

For environmental data measured by a variety of sensors and compiled from various sources, practitioners need tools that facilitate data access and data analysis. Data are often organized in formats that are incompatible with each other and that prevent full data integration. Furthermore, analyses of these data are hampered by the inadequate mechanisms for storage and organization. Ideally, data should be centrally housed and organized in an intuitive structure with established patterns for analyses. However, in reality, the data are often scattered in multiple files without uniform structure that must be transferred between users and called individually and manually for each analysis. This effort describes a process for compiling environmental data into a single, central database that can be accessed for analyses. We use the Logan River watershed and observed water level, discharge, specific conductance, and temperature as a test case. Of interest is analysis of flow partitioning. We formatted data files and organized them into a hierarchy, and we developed scripts that import the data to a database with structure designed for hydrologic time series data. Scripts access the populated database to determine baseflow separation, flow balance, and mass balance and visualize the results. The analyses were compiled into a package of scripts in Python, which can be modified and run by scientists and researchers to determine gains and losses in reaches of interest. To facilitate reproducibility, the database and associated scripts were shared to HydroShare as Jupyter Notebooks so that any user can access the data and perform the analyses, which facilitates standardization of these operations.

针对由各类传感器测量并从多种来源汇集的环境数据，实践者亟需能够促进数据访问与数据分析的工具。数据通常以彼此不兼容的格式组织，阻碍了数据的全面整合。此外，这些数据的分析受到存储和组织机制不足的制约。理想情况下，数据应集中存储，并以直观的结构组织，并建立分析的标准模式。然而，在实际操作中，数据往往分散于多个文件中，缺乏统一的结构，需要在不同用户间进行转移，且需手动逐一调用以进行每项分析。本描述旨在阐述将环境数据整合至单一中央数据库的过程，以便于分析之用。本研究以洛根河流域及其观测到的水位、流量、比电导率和温度作为测试案例。特别关注的是流量分配分析。我们对数据文件进行了格式化，并按层级组织，同时开发了将数据导入专为水文时间序列数据设计的数据库的脚本。脚本访问已填充的数据库，以确定基流分离、流量平衡和质量平衡，并可视化结果。分析结果被汇总成Python脚本的包，科学家和研究人员可据此修改和运行，以确定目标河段中的盈亏情况。为促进可重复性，数据库和相关脚本以Jupyter Notebooks的形式分享至HydroShare，使得任何用户均可访问数据并执行分析，从而促进了这些操作的标准化。