大数据湖中数据质量管控与数据治理实验数据

针对致景纺织企业业务系统中机器状态、ERP、MES、CRM、SCM相关的数据表，通过从数据库导出数据文件，通过大数据湖接口上传。数据采集于2024年7月15日，四川宜宾致景智造园。包含大数据湖中验证基于动态贝叶斯网络的离线大数据湖治理技术和多元数据管控机制的动态数据质量管控技术的数据，包括织机状态数据、ERP数据、MES数据、CRM数据、SCM数据，每种类型的数据包含准确数据和待清洗数据，支撑质量检测和状态监控中的数据保障和治理功能。本数据集为csv文件，可通过Python程序、MySQL数据库等进行读取或导入。每个数据文件夹包含dirty个clean两个数据表，分别代表待清洗的脏数据和已清洗的干净数据。数据清洗算法采用BClean[1]，通过Python分别读取dirty和clean两个数据集，进行模型的生成，最后对dirty数据进行清洗并生成repaired数据。以致景ERP数据为例，需要定义一些用户约束，以json的格式编写，其具体内容为：每个字段中数据的类型、最大长度、是否允许程序更改、是否满足正则表达式等。Targeting the data tables related to machine status, ERP, MES, CRM and SCM in the business system of Zhijing Textile Enterprise, this dataset is constructed by exporting data files from the database and uploading them via the big data lake interface. The data was collected on July 15, 2024 at Zhijing Intelligent Manufacturing Park in Yibin, Sichuan Province. It contains data used to validate dynamic data quality control technologies based on dynamic Bayesian network-driven offline big data lake governance technologies and multi-dimensional data management mechanisms in the big data lake, including loom status data, ERP data, MES data, CRM data and SCM data. Each type of data includes verified clean data and uncleaned dirty data, which supports data guarantee and governance functions in quality inspection and status monitoring. This dataset is provided in CSV format and can be read or imported via Python programs, MySQL databases and other tools. Each data folder contains two data tables: `dirty` and `clean`, which represent the uncleaned dirty data and preprocessed clean data respectively. The data cleaning algorithm adopts BClean [1]. Python is used to separately read the dirty and clean datasets to generate the model, and finally clean the dirty data to generate repaired data. Taking Zhijing ERP data as an example, some user constraints need to be defined in JSON format, including the data type, maximum length, whether program modification is allowed, whether it meets regular expressions and other specific requirements for each field.

搜集汇总

背景与挑战

背景概述

该数据集来自四川宜宾致景智造园纺织企业业务系统，采集于2024年7月15日，包含织机状态、ERP、MES、CRM和SCM等csv格式数据，旨在验证大数据湖中的数据治理与质量管控技术。数据分为待清洗和已清洗两部分，支持通过Python和MySQL进行读取和处理，应用于离散行业的质量检测与状态监控场景。