ONE DATA Data Sience Workflows

The ONE DATA data science workflow dataset ODDS-full comprises 815 unique workflows in temporally ordered versions. <br> A version of a workflow describes its evolution over time, so whenever a workflow is altered meaningfully, a new version of this respective workflow is persisted. <br> Overall, 16035 versions are available. The ODDS-full workflows represent machine learning workflows expressed as node-heterogeneous DAGs with 156 different node types.<br> These node types represent various kinds of processing steps of a general machine learning workflow and are grouped into 5 categories, which are listed below. <strong>Load</strong> Processors for loading or generating data (e.g. via a random number generator). <strong>Save</strong> Processors for persisting data (possible in various data formats, via external connections or as a contained result within the ONE DATA platform) or for providing data to other places as a service. <strong>Transformation</strong> Processors for altering and adapting data. This includes e.g. database-like operations such as renaming columns or joining tables as well as fully fledged dataset queries. <strong>Quantitative Methods</strong> Various aggregation or correlation analysis, bucketing, and simple forecasting. <strong>Advanced Methods</strong> Advanced machine learning algorithms such as BNN or Linear Regression. Also includes special meta processors that for example allow the execution of external workflows within the original workflow. Any metadata beyond the structure and node types of a workflow has been removed for anonymization purposes ODDS, a filtered variant, which enforces weak connectedness and only contains workflows with at least 5 different versions and 5 nodes, is available as the default version for supervised and unsupvervised learning. Workflows are served as JSON node-link graphs via <em>networkx.</em> They can be loaded into python as follows: <pre><code class="language-python">import pandas as pd import networkx as nx import json with open('ODDS.json', 'r') as f: graphs = pd.Series(list(map(nx.node_link_graph, json.load(f)['graphs'])))</code></pre>ONE DATA数据科学工作流数据集ODDS-full（ODDS-full）包含815个唯一工作流，每个工作流均拥有按时间顺序排列的多个版本。 工作流的每个版本均记录了其随时间的演进过程：每当工作流发生有意义的变更时，该工作流的新版本就会被持久化存储。 整体而言，该数据集共包含16035个工作流版本。 ODDS-full中的工作流均为机器学习工作流，以包含156种不同节点类型的节点异质有向无环图（node-heterogeneous Directed Acyclic Graph，DAG）形式表示。这些节点类型对应通用机器学习工作流中的各类处理步骤，共分为5大类，具体如下： **加载（Load）**：用于加载或生成数据的处理器（例如通过随机数生成器）。 **保存（Save）**：用于持久化存储数据（可通过外部连接或以独立结果的形式存储在ONE DATA平台内），或作为服务向其他位置提供数据的处理器。 **转换（Transformation）**：用于修改和适配数据的处理器，涵盖类数据库操作（例如重命名列、连接表）以及完整的数据集查询等功能。 **定量方法（Quantitative Methods）**：各类聚合或相关性分析、分箱以及简单预测任务的处理器。 **高级方法（Advanced Methods）**：各类高级机器学习算法（例如贝叶斯神经网络（BNN）、线性回归等），还包含特殊元处理器，例如允许在原始工作流中执行外部工作流的处理器。 为实现匿名化，所有超出工作流结构与节点类型范畴的元数据均已被移除。ODDS是经过筛选的变体数据集，其要求工作流满足弱连通性，且仅包含至少拥有5个版本与5个节点的工作流，可作为监督学习与无监督学习任务的默认数据集版本。 工作流以JSON节点链接图（JSON node-link graph）的形式通过NetworkX（networkx）提供，可通过以下代码将其加载至Python环境中： <pre><code class="language-python">import pandas as pd import networkx as nx import json with open('ODDS.json', 'r') as f: graphs = pd.Series(list(map(nx.node_link_graph, json.load(f)['graphs'])))</code></pre>