Pricing-Driven Resource Allocation in the Computing Continuum – Laboratory Package

This repository contains the implementation used to study pricing-driven resource allocation in the computing continuum. The workflow generates topology-specific pricing models, creates constrained problem instances, and delegates optimization to PRIME through its REST API. The project is intended for research-grade experimentation and reproducibility. GitHub URL: https://anonymous.4open.science/r/services-allocation Table of Contents 1. Project Structure2. How to Reproduce the Experiment3. API of pricing_driven_resource_allocation4. Data and Outputs5. License & Disclaimer Project Structure The repository is organized as follows (main elements only): services-allocation/├── config/│ └── experiment_configuration.yml # Scenario definitions (small/medium/large)├── docker-compose.yml # PRIME analysis API service (port 3000)├── evaluation.ipynb # End-to-end experimental pipeline├── eua-dataset/│ ├── edge-servers/ # Input edge-node datasets│ └── users/ # Input user-location datasets├── iPricing/│ ├── iPricing.proto # Pricing model schema│ └── model/ # Generated Python protobuf module├── pricing_driven_resource_allocation/ # Core Python package│ ├── __init__.py│ ├── optimize.py # PRIME API client and polling loop│ ├── dataset/│ │ ├── load.py # Dataset loading utilities│ │ ├── transform.py # Filtering and resource assignment│ │ └── save_results.py # Results persistence (CSV)│ ├── generators/│ │ ├── topology.py # Topology synthesis per scenario│ │ ├── pricing.py # Pricing YAML generation│ │ ├── problem_instance.py # Request-constrained instance construction│ │ ├── client_demand.py # Demand modeling by application class│ │ └── request.py # Request payload builder│ └── utils/│ ├── geometrical_utils.py # Spatial computations│ └── yaml_utils.py # YAML <-> protobuf conversion helpers├── results/│ ├── results.csv # Aggregated optimization outcomes│ └── figures/ # Publication-ready plots├── synthetic-dataset/│ ├── data/│ └── synthetic-topologies/ # 9600 generated topologies and instances├── requirements.txt├── setup.py└── README.md How to Reproduce the Experiment > [!WARNING] > The execution of this experiment can be computationally intensive and may require several hours, particularly when running the large-scale scenario. Before starting the procedure, ensure that adequate computational resources and sufficient uninterrupted execution time are available.>> For reference, the experiment was conducted on a workstation equipped with an Apple Silicon M4 Pro processor and 24 gigabytes of main memory, where the complete execution required approximately **ten hours**. 1. Prerequisites - Python 3.10+ recommended- Docker and Docker Compose- Protocol Buffers compiler (`protoc`)- Jupyter (to execute `evaluation.ipynb`) 2. Clone and install To clone the repository, use: ```bashgit clone <repository-url>``` or download the zip file from the anonymous repository. ```bashcd resource-allocationpython -m venv .venvsource .venv/bin/activatepip install --upgrade pippip install -r requirements.txtpip install -e .``` 3. Confirm port 3000 is available Before launching PRIME, verify that no process is currently bound to TCP port 3000: ```bashlsof -i :3000``` If the command returns any process, release the port before continuing. 4. Start PRIME from the project root From the repository root, launch the PRIME service exactly as follows: ```bashdocker-compose up -d``` Then verify health: ```bashcurl http://localhost:3000/health``` The notebook uses `PRIME_INSTANCE_URL = "http://localhost:3000/api/v1/"`, therefore PRIME must be reachable on port 3000. If you run PRIME on any other port, remember to update the URL in the first cell of `evaluation.ipynb` accordingly. 5. Execute the experiment pipeline Run the notebook and execute all cells in order: ```bashjupyter notebook evaluation.ipynb``` Pipeline stages implemented in the notebook: 1. Initialize constants, paths, offer configuration, and resources.2. Load and validate `config/experiment_configuration.yml`.3. Load and preprocess the EUA edge dataset.4. Generate topologies for each scenario and repetition.5. Build pricing files and scenario-specific problem instances.6. Invoke PRIME optimization through `pdsa.optimize(...)`.7. Persist execution metrics in `results/results.csv` and generate figures in `results/figures/`. 6. Stop services after completion ```bashdocker-compose down``` API of pricing_driven_resource_allocation The package exposes four public namespaces at the top level: - `pdsa.dataset`- `pdsa.generators`- `pdsa.utils`- `pdsa.optimize` Top-level API ```pythonimport pricing_driven_service_allocation as pdsa pdsa.optimize(...)pdsa.dataset.*pdsa.generators.*pdsa.utils.*``` `pdsa.dataset` - `load_devices_dataframe(path: str) -> pandas.DataFrame` Loads the raw edge device CSV and standardizes column names.- `load_client_locations_dataframe(path: str) -> pandas.DataFrame` Loads and normalizes client geolocation data.- `filter_devices_by_vendors(devices_df: pandas.DataFrame, vendors_to_consider: list) -> pandas.DataFrame` Filters devices by provider tokens in device names and adds a normalized `provider` field.- `assign_device_resources(df: pandas.DataFrame, config: dict | None = None, seed: int | None = None) -> pandas.DataFrame` Assigns capacities, prices, global groups, and device classes according to configurable stochastic rules.- `save_results_to_csv(result_obj: dict, scenario_id: str, RESULTS_DIR: str, filename: str = "results.csv", include_filter: bool = True) -> None` Stores optimization outcomes and filter metadata in CSV format. `pdsa.generators` - `topology(...) -> tuple[pandas.DataFrame, str]` Creates a topology constrained by center, radius, providers, and device count; writes `devices.csv`, `metadata.json`, and `map.html`. Core signature: ```python pdsa.generators.topology( lat: float, long: float, rad: float, devices_df: pandas.DataFrame, topologies_result_dir: str, resources_to_consider: list[str], number_of_providers: int | None = None, allowed_groups: list[int] | None = None, number_of_devices: int | None = None, center_elevation: float = 0.0, options: dict | None = None, ) ``` - `pricing_from_topology(...) -> str` Converts a generated topology into a pricing YAML instance (`pricing.yml`).- `compatible_provider_groups_from_offer(topology_offer: dict) -> list[list[str]]` Computes compatible provider groups from exclusion constraints.- `problem_instance(instance_pricing, request: dict, topologies_result_dir: str, unlimited_value: int = 100000000, options: dict = ...) -> tuple` Generates a request-constrained pricing instance and a solver filter.- `request(topology_demand: dict, topology_request: dict, users_demand: dict, resources_to_consider: list[str], currency: str = "USD", resource_mapping: dict | None = None) -> dict` Builds normalized request payloads for problem-instance generation.- `client_demand.calculate_resources(...) -> dict` Estimates resource demand from user volume and application behavior profiles. `pdsa.utils` - `yaml_to_pricing_proto(yaml_path: str, message_type)` Parses pricing YAML into protobuf objects.- `pricing_proto_to_yaml(pricing_obj, yaml_path: str, options: dict | None = None) -> None` Serializes protobuf pricing instances into YAML.- `find_identical_addons(pricing_obj) -> list[tuple[str, str]]` Detects structurally identical add-ons.- `haversine(...) -> float`- `distance_3d(...) -> float`- `point_in_polygon(...) -> bool`- `distance_to_farthest_edge(...) -> float` `pdsa.optimize` - `optimize(prime_instance_url: str, pricing_instance_path: str, request: dict, poll_interval_seconds: float = 0.1, timeout_seconds: float | None = 600.0, session: requests.Session | None = None) -> dict` Behavior: 1. Submits a multipart optimization job to `POST {prime_instance_url}/pricing/analysis`.2. Polls `GET {prime_instance_url}/pricing/analysis/{jobId}` until terminal status.3. Returns the final payload (`COMPLETED` or `FAILED`). ## Data and Outputs - Input datasets: - `eua-dataset/edge-servers/site.csv` - `eua-dataset/users/users-aus.csv`- Scenario specification: - `config/experiment_configuration.yml`- Generated artifacts: - `synthetic-dataset/synthetic-topologies/<topology_id>/devices.csv` - `synthetic-dataset/synthetic-topologies/<topology_id>/pricing.yml` - `synthetic-dataset/synthetic-topologies/<topology_id>/problem_instance_pricing.yml` - `results/results.csv` - `results/figures/*.png` ⚠️ Disclaimer & License LICENSE This project is licensed under the MIT License. See [LICENSE](./LICENSE) for details DISCLAIMER This tool is part of ongoing research by the XXXX in pricing-driven development and operation. It is in an **early stage** and is not intended for production use. The XXXX does not accept responsibility for any issues or damages that may arise from its use in real-world environments本仓库包含用于研究计算连续体（computing continuum）中定价驱动型资源分配的实现代码。本工作流可生成针对特定拓扑的定价模型、构建带约束的问题实例，并通过REST API（REST API）将优化任务委托给PRIME（PRIME）工具。本项目专为科研级实验与可复现性设计。 GitHub 仓库地址：https://anonymous.4open.science/r/services-allocation 目录 1. 项目结构 2. 实验复现指南 3. pricing_driven_resource_allocation 模块API 4. 数据与输出 5. 许可证与免责声明 ## 项目结构 本仓库的组织形式如下（仅列出核心元素）： services-allocation/ ├── config/ │ └── experiment_configuration.yml # 场景定义文件（涵盖小/中/大型场景） ├── docker-compose.yml # PRIME分析API服务（端口3000） ├── evaluation.ipynb # 端到端实验流水线 ├── eua-dataset/ │ ├── edge-servers/ # 边缘节点输入数据集 │ └── users/ # 用户位置输入数据集 ├── iPricing/ │ ├── iPricing.proto # 定价模型模式定义 │ └── model/ # 生成的Python Protobuf模块 ├── pricing_driven_resource_allocation/ # 核心Python包 │ ├── __init__.py │ ├── optimize.py # PRIME API客户端与轮询循环 │ ├── dataset/ │ │ ├── load.py # 数据集加载工具 │ │ ├── transform.py # 数据过滤与资源分配 │ │ └── save_results.py # 结果持久化（CSV格式） │ ├── generators/ │ │ ├── topology.py # 按场景合成拓扑 │ │ ├── pricing.py # 定价YAML文件生成 │ │ ├── problem_instance.py # 构建带约束的请求实例 │ │ ├── client_demand.py # 按应用类别建模需求 │ │ └── request.py # 请求负载构建器 │ └── utils/ │ ├── geometrical_utils.py # 空间计算工具 │ └── yaml_utils.py # YAML与Protobuf转换辅助工具 ├── results/ │ ├── results.csv # 聚合优化结果 │ └── figures/ # 可用于发表的可视化图表 ├── synthetic-dataset/ │ ├── data/ │ └── synthetic-topologies/ # 9600个生成的拓扑与实例 ├── requirements.txt ├── setup.py └── README.md ## 实验复现指南 ⚠️ 警告 本实验的计算量较大，运行耗时可能较长，在运行大型场景时尤为明显。开始实验前，请确保具备充足的计算资源与不间断的运行时间。 参考案例：本实验在搭载Apple Silicon M4 Pro处理器与24GB主内存的工作站上完成，完整运行耗时约**十小时**。 1. 前置依赖 - 推荐使用Python 3.10及以上版本 - Docker与Docker Compose - 协议缓冲区（Protocol Buffers）编译器（`protoc`） - Jupyter（Jupyter，用于运行`evaluation.ipynb`） 2. 克隆与安装 克隆仓库的命令如下： bash git clone <仓库地址> 或从匿名仓库下载压缩包。 进入仓库目录并配置环境： bash cd resource-allocation python -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install -r requirements.txt pip install -e . 3. 确认3000端口未被占用 在启动PRIME前，请检查TCP 3000端口是否未被其他进程占用： bash lsof -i :3000 若该命令返回已有进程占用该端口，请先释放端口后再继续。 4. 从项目根目录启动PRIME 在仓库根目录下，执行以下命令启动PRIME服务： bash docker-compose up -d 随后验证服务健康状态： bash curl http://localhost:3000/health 本实验的Jupyter Notebook中预设了`PRIME_INSTANCE_URL = "http://localhost:3000/api/v1/"`，因此PRIME服务需监听3000端口。若您将PRIME运行在其他端口，请务必修改`evaluation.ipynb`首个代码单元格中的对应URL。 5. 运行实验流水线 按顺序执行Jupyter Notebook中的所有代码单元格： bash jupyter notebook evaluation.ipynb 本Notebook实现的流水线步骤如下： 1. 初始化常量、路径、服务配置与资源 2. 加载并校验`config/experiment_configuration.yml` 3. 加载并预处理EUA边缘数据集 4. 为每个场景与重复实验生成拓扑 5. 构建定价文件与场景专属的问题实例 6. 通过`pdsa.optimize(...)`调用PRIME优化任务 7. 将执行指标持久化至`results/results.csv`，并在`results/figures/`中生成可视化图表 6. 实验结束后停止服务 bash docker-compose down ## pricing_driven_resource_allocation 模块API 本包对外暴露四个顶层公共命名空间： - `pdsa.dataset` - `pdsa.generators` - `pdsa.utils` - `pdsa.optimize` ### 顶层API示例 python import pricing_driven_service_allocation as pdsa pdsa.optimize(...) pdsa.dataset.* pdsa.generators.* pdsa.utils.* ### `pdsa.dataset` - `load_devices_dataframe(path: str) -> pandas.DataFrame`：加载原始边缘设备CSV文件并标准化列名，返回pandas数据框（pandas.DataFrame）。 - `load_client_locations_dataframe(path: str) -> pandas.DataFrame`：加载并规范化客户端地理定位数据，返回pandas数据框（pandas.DataFrame）。 - `filter_devices_by_vendors(devices_df: pandas.DataFrame, vendors_to_consider: list) -> pandas.DataFrame`：按设备名称中的提供商标识过滤设备，并添加规范化的`provider`字段，返回pandas数据框（pandas.DataFrame）。 - `assign_device_resources(df: pandas.DataFrame, config: dict | None = None, seed: int | None = None) -> pandas.DataFrame`：根据可配置的随机规则为设备分配容量、定价、全局分组与设备类别，返回pandas数据框（pandas.DataFrame）。 - `save_results_to_csv(result_obj: dict, scenario_id: str, RESULTS_DIR: str, filename: str = "results.csv", include_filter: bool = True) -> None`：以CSV格式存储优化结果与过滤元数据。 ### `pdsa.generators` - `topology(...) -> tuple[pandas.DataFrame, str]`：创建受中心坐标、半径、提供商与设备数量约束的拓扑，并写入`devices.csv`、`metadata.json`与`map.html`。 核心签名如下： python pdsa.generators.topology( lat: float, long: float, rad: float, devices_df: pandas.DataFrame, topologies_result_dir: str, resources_to_consider: list[str], number_of_providers: int | None = None, allowed_groups: list[int] | None = None, number_of_devices: int | None = None, center_elevation: float = 0.0, options: dict | None = None, ) - `pricing_from_topology(...) -> str`：将生成的拓扑转换为定价YAML实例（`pricing.yml`）。 - `compatible_provider_groups_from_offer(topology_offer: dict) -> list[list[str]]`：从排除约束中计算兼容的提供商分组。 - `problem_instance(instance_pricing, request: dict, topologies_result_dir: str, unlimited_value: int = 100000000, options: dict = ...) -> tuple`：生成带请求约束的定价实例与求解器过滤器。 - `request(topology_demand: dict, topology_request: dict, users_demand: dict, resources_to_consider: list[str], currency: str = "USD", resource_mapping: dict | None = None) -> dict`：为问题实例生成构建规范化的请求负载。 - `client_demand.calculate_resources(...) -> dict`：根据用户规模与应用行为概况估算资源需求。 ### `pdsa.utils` - `yaml_to_pricing_proto(yaml_path: str, message_type)`：将定价YAML解析为Protobuf对象。 - `pricing_proto_to_yaml(pricing_obj, yaml_path: str, options: dict | None = None) -> None`：将Protobuf定价实例序列化为YAML文件。 - `find_identical_addons(pricing_obj) -> list[tuple[str, str]]`：检测结构完全相同的附加组件。 - `haversine(...) -> float`：计算球面两点间距离（哈弗辛公式）。 - `distance_3d(...) -> float`：计算三维空间两点间距离。 - `point_in_polygon(...) -> bool`：判断点是否在多边形内部。 - `distance_to_farthest_edge(...) -> float`：计算点到最远边缘的距离。 ### `pdsa.optimize` - `optimize(prime_instance_url: str, pricing_instance_path: str, request: dict, poll_interval_seconds: float = 0.1, timeout_seconds: float | None = 600.0, session: requests.Session | None = None) -> dict` #### 行为说明： 1. 向`POST {prime_instance_url}/pricing/analysis`提交多部分优化任务。 2. 轮询`GET {prime_instance_url}/pricing/analysis/{jobId}`直至任务进入终端状态。 3. 返回最终负载（`COMPLETED`或`FAILED`）。 ## 数据与输出 - 输入数据集： - `eua-dataset/edge-servers/site.csv` - `eua-dataset/users/users-aus.csv` - 场景配置文件： - `config/experiment_configuration.yml` - 生成的产物： - `synthetic-dataset/synthetic-topologies/<topology_id>/devices.csv` - `synthetic-dataset/synthetic-topologies/<topology_id>/pricing.yml` - `synthetic-dataset/synthetic-topologies/<topology_id>/problem_instance_pricing.yml` - `results/results.csv` - `results/figures/*.png` ⚠️ 免责声明与许可证 ### 许可证 本项目采用MIT许可证授权，详细信息请参见[LICENSE](./LICENSE)文件。 ### 免责声明 本工具为XXXX团队在定价驱动型开发与运维领域的在研项目，目前处于**早期阶段**，不适合用于生产环境。XXXX团队不对其在实际生产环境中使用所引发的任何问题或损失承担责任。