Supplementary information for Hierarchical game-theoretic and reinforcement learning framework for computational offloading in UAV-enabled mobile edge computing networks with multiple service providers

Article abstractWe present a novel game-theoretic (GT) and reinforcement learning (RL) framework for computational offloading in the mobile edge computing (MEC) network operated by multiple service providers (SPs). The network is formed by MEC servers installed at stationary base stations (BSs) and unmanned aerial vehicles (UAVs) deployed as quasi-stationary BSs. Since computing powers of MEC servers are limited, the BSs in proximity can form coalitions with shared data processing resources to serve their users more efficiently. However, as BSs can be privately owned or controlled by different SPs, in any coalition, the BSs: 1) take only the actions that maximize their long-term payoffs and 2) do not coordinate their actions with other BSs in the coalition. That is, inside each coalition, BSs act in an independent and self-interested manner. Therefore, the interactions among BSs cannot be described by conventional coalitional games. Instead, the network operation is modeled by a two-level hierarchical model. The upper level is a cooperative game that defines the process of coalition formation. The lower level comprises the set of noncooperative subgames to represent a self-interested and independent behavior of BSs in coalitions. To enable each BS to select a coalition and decide on its action maximizing its long-term payoff, we propose two algorithms that combine coalition formation with RL and prove that these algorithms converge to the states where the coalitional structure is strongly stable and the strategies of BSs are in the mixed-strategy Nash equilibrium (NE).
© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

论文摘要
本文提出了一种新颖的博弈论（GT）与强化学习（RL）框架，用于多服务提供商（SPs）运营的移动边缘计算（MEC）网络中的计算卸载任务。该网络由部署在固定基站（BSs）上的移动边缘计算服务器，以及作为准静止基站部署的无人机（UAVs）共同构建。由于移动边缘计算服务器的计算能力有限，邻近基站可通过组建联盟共享数据处理资源，以更高效地为用户提供服务。然而，由于基站可由不同服务提供商私有所有或管控，在任意联盟中，基站均需满足两个条件：1）仅采取能最大化自身长期收益的行动；2）不会与联盟内其他基站协调行动。换言之，在每个联盟内部，基站均以独立自利的方式行事。因此，传统联盟博弈无法刻画基站间的交互关系。为此，我们采用双层层级模型对网络运行机制进行建模：上层为合作博弈，用于定义联盟组建过程；下层则包含一系列非合作子博弈，用以表征联盟内基站的自利独立行为。为使每个基站能够选择联盟并确定最大化其长期收益的行动，我们提出了两种将联盟组建与强化学习相结合的算法，并证明了这些算法可收敛至如下状态：联盟结构具备强稳定性，且基站的策略处于混合策略纳什均衡（NE）。
© 2019 IEEE。允许个人使用本材料。若需在任何当前或未来媒介中进行其他使用，包括为广告或促销目的重印/再发布本材料、创建新的集体作品、转售或重新分发至服务器或列表，或在其他作品中复用本作品的任何受版权保护的组件，均需获得IEEE的许可。