syslrn: Learning What to Monitor for Efficient Anomaly Detection [Dataset]

This repository includes the dataset for the paper:

D. Sanvito, G. Siracusano, S. Santhanam, R. Gonzalez, R. Bifulco
syslrn: Learning What to Monitor for Efficient Anomaly Detection
ACM EuroMLSys 2022

The dataset contains two directories at the root level:

raw_dataset

processed_dataset

Each folder in the raw_dataset directory contains the raw monitoring data used to generate the graph associated to a single experiment together with additional metadata files.
Each folder in the processed_dataset directory contains the graph associated to a single experiment as a set of three CSV files: two for the graph edges (pid_childof_pid_df.csv and pid_speakswith_pid_df.csv) and one for the graph nodes (proc_df.csv).
We provide below a code snippet to parse a graph from processed_dataset directory.

In both folders the name of each sub-folder is based on the following schema: [SCENARIO]_[W]wl/test_[TEST_ID] where:

[SCENARIO] reports the target component for the failure injection (cinder_failure, neutron_failure, nova_failure). ff  indicates instead a failure-free execution

[W] reports the number of concurrent workloads

[TEST_ID] reports the ID of the specific failure scenario injected (same ID selected by the OpenStack failure injection framework [1] )

Each experiment includes the following data in the raw_dataset sub-folders:

audit_raw_logs_[TEST_ID]/: raw audit monitoring data

bpf_tools_[TEST_ID]/: raw ebpf tools monitoring data

instance-[INSTANCE_ID]/: workload-specific metadata files, e.g. stdout/stderr (generated by the OpenStack failure injection framework [1] )

logs_workload_[TEST_ID]/: OpenStack application logs

perf_tools_[TEST_ID]/: raw perf tools monitoring data

audit_filtered_[TEST_ID].log: audit data pre-processed by ausearch (e.g. numerical entities are resolved to symbols)

failure_[TEST_ID].info: metadata information about the specific failure scenario (generated by the OpenStack failure injection framework [1] )

timestamps_[TEST_ID]: timing information

[1] D. Cotroneo, L. De Simone, P. Liguori, R. Natella, N. Bidokhti - How Bad Can a Bug Get? An Empirical Analysis of Software Failures in the OpenStack Cloud Computing Platform [ACM ESEC/FSE 2019]

Example: parsing a graph from processed_dataset directory

import pandas as pd
import networkx as nx

def parse_csv(path):
processes_df = pd.read_csv('%sproc_df.csv' % path, index_col=0).reset_index(drop=True)

speakswith_edges_df = pd.read_csv('%spid_speakswith_pid_df.csv' % path, index_col=0)
speakswith_edges_df['type'] = 'speaksWith'

childof_edges_df = pd.read_csv('%spid_childof_pid_df.csv' % path, index_col=0)
childof_edges_df['type'] = 'childOf'

return processes_df, pd.concat([speakswith_edges_df, childof_edges_df], ignore_index=True)

def make_graph(nodes_df, edges_df):
G = nx.MultiGraph()

for _, node in nodes_df.iterrows():
G.add_node(node.pid, **node)

for _, edge in edges_df.iterrows():
G.add_edge(edge.pid1, edge.pid2, type=edge.type)

return G

PATH = 'processed_dataset/ff_1wl/test_1/'
nodes_df, edges_df = parse_csv(PATH)
G = make_graph(nodes_df, edges_df)
nx.draw_networkx(G, node_size=10, with_labels=False)

If you use this dataset for your research, please cite the following paper:

@inproceedings{sanvito2022syslrn,
title={syslrn: Learning What to Monitor for Efficient Anomaly Detection},
author={Sanvito, Davide and Siracusano, Giuseppe and Santhanam, Sharan and Gonzalez, Roberto and Bifulco, Roberto},
booktitle={2nd European Workshop on Machine Learning and Systems (EuroMLSys '22)},
year={2022},
address = {Rennes, France},
publisher = {ACM},
month = apr,
}