Correction workflow and spatial database model of Aquopts - A Hydrological Optical Data Processing System

In order to improve the capacity of storage, exploration and processing of sensor data, a spatial DBMS was used and the Aquopts system was implemented.

In field surveys using different sensors on the aquatic environment, the existence of spatial attributes in the dataset is common, motivating the adoption of PostgreSQL and its spatial extension PostGIS. To enable the insertion of new data sets as well as new devices and sensing equipment, the database was modeled to support updates and provide structures for storing all the data collected in the field campaigns in conjunction with other possible future data sources. The database model provides resources to manage spatial and temporal data and allows flexibility to select and filter the dataset.

The data model ensures the storage integrity of the information related to the samplings performed during the field survey in an architecture that benefits the organization and management of the data. However, in addition to the storage specified on the data model, there are several procedures that need to be applied to the data to prepare it for analysis. Some validations are important to identify spurious data that may represent important sources of information about data quality. Other corrections are essential to tweak the data and eliminate undesirable effects. Some equations can be used to produce other factors that can be obtained from the combination of attributes. In general, the processing steps comprise a cycle of important operations that are directly related to the characteristics of the data set. Considering the data of the sensors stored in the database, an interactive prototype system, named Aquopts, was developed to perform the necessary standardization and basic corrections and produce useful data for analysis, according to the correction methods known in the literature.

The system provides resources for the analyst to automate the process of reading, inserting, integrating, interpolating, correcting, and other calculations that are always repeated after exporting field campaign data and producing new data sets. All operations and processing required for data integration and correction have been implemented from the PHP and Python language and are available from a Web interface, which can be accessed from any computer connected to the internet. The data access cab be access online (http://sertie.fct.unesp.br/aquopts), but the resources are restricted by registration and permissions for each user. After their identification, the system evaluates the access permissions and makes available the options of insertion of new datasets.

The source-code of the entire Aquopts system are available at: https://github.com/carmoafc/aquopts

The system and additional results were described on the official paper (under review)

为提升传感器数据的存储、探索与处理能力，本研究采用空间数据库管理系统（spatial DBMS）并实现了Aquopts系统。

在针对水生环境开展的多传感器野外调研中，数据集普遍包含空间属性，因此本研究选用PostgreSQL及其空间扩展插件PostGIS。为支持新增数据集、设备与传感仪器的接入，该数据库模型设计为可更新架构，并提供存储结构以保存野外作业采集的全部数据，同时兼容未来可能接入的其他数据源。该数据库模型可管理时空数据，并支持灵活选择与过滤数据集。

该数据模型可保障野外采样相关信息的存储完整性，其架构有助于数据的组织与管理。然而，除数据模型规定的存储功能外，还需通过多项处理流程为后续分析做好数据准备：部分验证流程可识别伪异常数据，此类数据实则可能蕴含影响数据质量的关键信息；另有部分校正流程可优化数据、消除不良影响。此外，还可通过属性组合运算生成衍生因子。总体而言，处理流程构成一个关键操作循环，其细节与数据集的特性直接相关。

针对存储于数据库中的传感器数据，本研究开发了一款名为Aquopts的交互式原型系统，可依据文献中已有的校正方法，完成必要的标准化与基础校正流程，生成可供分析的有效数据。

该系统可为分析人员提供自动化处理能力，涵盖野外调研数据导出后的常规读取、插入、集成、插值、校正及其他重复计算流程。所有数据集成与校正所需的操作均基于PHP与Python语言实现，并通过Web界面提供服务，可通过任意联网计算机访问。该系统支持在线访问，地址为：http://sertie.fct.unesp.br/aquopts，但所有功能均需通过用户注册与权限验证后方可使用。系统会在用户身份验证通过后，评估其访问权限，并开放新增数据集的相关操作选项。

整个Aquopts系统的源代码已公开于：https://github.com/carmoafc/aquopts

该系统与补充实验结果已在投稿待审的正式学术论文中进行阐述。