HyDuo

This repository contains the source code, datasets, and evaluation artifacts for HyDuo, a software-defined, access-aware hybrid storage system tailored for stateful serverless computing. HyDuo bridges the gap between ultra-low-latency in-memory systems and high-latency object stores by dynamically orchestrating intermediate data across heterogeneous storage tiers (DRAM, NVMe, SSD, HDD). It uses a Generalized Assignment Problem (GAP) solver combined with online access pattern prediction (frequency, interval, periodicity, and inter-object correlation) to minimize end-to-end latency under a strict user-defined cost budget. 📑 Table of Contents Repository Structure Prerequisites System Architecture Installation & Deployment Datasets and Workloads Running the Evaluation Plotting and Reproducing Figures Citation 🛠 Prerequisites To deploy and test HyDuo on a physical cluster, you will need: Kubernetes Cluster (v1.3.1 or higher) OpenFaaS (v0.27 or higher) Storage Backend: MinIO (used to simulate S3/HDD tiers) and local RAM disks/NVMe mounts for high-performance tiers. Message Broker: Kafka or NATS (for control-plane event streaming). Python 3.8+ (for the Placement Engine solver and plotting scripts). Go 1.18+ (for building the sidecar). 🏗 System Architecture HyDuo operates entirely transparently to the serverless application logic. Data Plane: A lightweight Sidecar intercepts all reads/writes from the function pods, forwarding access events to the Message Bus. Control Plane: The Profiler aggregates events and extracts statistical features. The Placement Engine uses an $\epsilon$-constraint GAP formulation to find the optimal data placement (minimizing latency subject to a $C_{max}$ budget). The Migration Executor safely moves blobs between tiers (DRAM $\leftrightarrow$ NVMe $\leftrightarrow$ SSD $\leftrightarrow$ HDD) without blocking function execution. 🚀 Installation & Deployment 1. Setup the Storage Tiers Configure your Kubernetes worker nodes with the appropriate labels for heterogeneous storage, then deploy the storage backends: Bash kubectl apply -f kubernetes-manifests/storage-tiers/ 2. Deploy the Control Plane Deploy the Kafka message bus, Metadata Store (Etcd/Postgres), and HyDuo control components: Bash kubectl apply -f kubernetes-manifests/control-plane/ 3. Configure OpenFaaS to inject the HyDuo Sidecar Apply the mutating webhook that automatically injects the hyduo-sidecar into any deployed OpenFaaS function: Bash kubectl apply -f kubernetes-manifests/webhook-injection.yaml 📊 Datasets and Workloads The datasets/ directory contains down-sampled traces and input files to reproduce the evaluation. The full datasets (tens of GBs) are hosted externally due to GitHub size limits. You can download the full datasets by running: Bash bash datasets/download_full_datasets.sh Supported Benchmarks: Video Analytics: ExCamera Distributed Compilation: gg_cmake Matrix Computation: NumPywren Machine Learning: CNN (CIFAR-10) Scientific Computing: Metaspace