Correction workflow and spatial database model of Aquopts - A Hydrological Optical Data Processing System

In order to improve the capacity of storage, exploration and processing of sensor data, a spatial DBMS was used and the Aquopts system was implemented. In field surveys using different sensors on the aquatic environment, the existence of spatial attributes in the dataset is common, motivating the adoption of PostgreSQL and its spatial extension PostGIS. To enable the insertion of new data sets as well as new devices and sensing equipment, the database was modeled to support updates and provide structures for storing all the data collected in the field campaigns in conjunction with other possible future data sources. The database model provides resources to manage spatial and temporal data and allows flexibility to select and filter the dataset. The data model ensures the storage integrity of the information related to the samplings performed during the field survey in an architecture that benefits the organization and management of the data. However, in addition to the storage specified on the data model, there are several procedures that need to be applied to the data to prepare it for analysis. Some validations are important to identify spurious data that may represent important sources of information about data quality. Other corrections are essential to tweak the data and eliminate undesirable effects. Some equations can be used to produce other factors that can be obtained from the combination of attributes. In general, the processing steps comprise a cycle of important operations that are directly related to the characteristics of the data set. Considering the data of the sensors stored in the database, an interactive prototype system, named Aquopts, was developed to perform the necessary standardization and basic corrections and produce useful data for analysis, according to the correction methods known in the literature. The system provides resources for the analyst to automate the process of reading, inserting, integrating, interpolating, correcting, and other calculations that are always repeated after exporting field campaign data and producing new data sets. All operations and processing required for data integration and correction have been implemented from the PHP and Python language and are available from a Web interface, which can be accessed from any computer connected to the internet. The data access cab be access online (http://sertie.fct.unesp.br/aquopts), but the resources are restricted by registration and permissions for each user. After their identification, the system evaluates the access permissions and makes available the options of insertion of new datasets. The source-code of the entire Aquopts system are available at: https://github.com/carmoafc/aquopts The system and additional results were described on the official paper (under review)为提升传感器数据的存储、探索与处理能力，本研究采用空间数据库管理系统（Spatial DBMS）并实现了Aquopts系统。 在针对水生环境开展的多传感器野外调查中，数据集普遍包含空间属性，因此选用PostgreSQL及其空间扩展插件PostGIS。为支持新增数据集、新设备与传感装置的接入，数据库在建模阶段兼顾了更新需求，并构建了可存储野外监测项目采集数据的结构，同时兼容未来可能接入的其他数据源。该数据库模型可管理时空数据，并支持灵活筛选与过滤数据集。 该数据模型保障了野外调查采样相关信息的存储完整性，其架构有利于数据的组织与管理。不过，除数据模型指定的存储功能外，还需通过多项处理流程对数据进行预处理以支撑后续数据分析：部分验证操作可用于识别虚假数据，而这类数据可为数据质量评估提供关键信息；另有部分校正操作可用于调整数据并消除不良影响。此外，还可通过方程运算由属性组合生成其他衍生因子。总体而言，数据处理流程构成了一套关键操作循环，其具体环节与数据集特性直接相关。针对存储于数据库中的传感器数据，本研究开发了一款名为Aquopts的交互式原型系统，可依据文献中已公开的校正方法，完成必要的标准化与基础校正操作，生成可供分析的有效数据。 该系统可为数据分析人员提供自动化流程支持，替代此前需在导出野外监测数据、生成新数据集后重复执行的读取、插入、整合、插值、校正及其他计算操作。所有数据整合与校正所需的操作与处理流程均基于PHP与Python语言实现，并通过Web界面对外提供服务，任何联网计算机均可访问该界面。该系统支持在线访问（访问地址：http://sertie.fct.unesp.br/aquopts），但所有功能均需通过用户注册与权限管控。用户完成身份验证后，系统会评估其访问权限，并开放新增数据集的相关操作选项。 Aquopts系统的完整源代码已开源至：https://github.com/carmoafc/aquopts 本系统与补充实验结果已收录于待审的正式学术论文中。