Deep Reinforcement Learning Based 3D-Trajectory Design and Task Offloading in UAV-enabled MEC System

In unmanned aerial vehicle (UAV) enabled mobile edge computing (MEC) system, rotary-wing UAV can be dispatched to fly close to ground terminals (GTs) to execute their offloaded tasks. This can extend GTs’ computing capability and save their energy cost. However, to enhance the energy efficiency of UAV propulsion, ensure successful completion of each GT's mission, and maintain a stable UAV-GT uplinks, it is crucial to design a rational UAV 3D trajectory and mission offloading strategy. To address this joint problem, We further build a deep reinforcement learning (DRL)-based framework in the UAV-enabled MEC system, which applies Double Deep Q-learning (D-DQN) and convex optimization techniques to jointly work out the near-optimal UAV 3D-trajectory and task offloading strategy among GTs in an online manner. With the proposed framework, the UAV-enabled MEC system can finish each GT task quickly and save the UAV mission time and propulsion energy effectively. Numerical results show that the DLR-based framework outperforms the benchmark solutions, e.g., In scenario 1, the throughput, the average data transferring rate through UAV-GT links, and the average energy efficiency of UAV compared to DQN are improved from 64.24Kb to 72.99Kb, 4.72Kbps to 20.75Kbps, 0.32bits/J to 0.36bits/J, respectively, and almost the same as the existing Travelling Salesman Problem (TSP) solution, making the energy efficient edge computing truly viable in UAV-enabled MEC system.

在搭载无人机（unmanned aerial vehicle, UAV）的移动边缘计算（mobile edge computing, MEC）系统中，可调度旋翼无人机（rotary-wing UAV）飞行至贴近地面终端（ground terminals, GTs）的位置，以执行地面终端卸载的计算任务。此举可拓展地面终端的计算能力并降低其能耗成本。然而，为提升无人机推进系统的能效、保障各地面终端任务顺利完成，并维持稳定的无人机-地面终端上行链路，设计合理的无人机三维轨迹与任务卸载策略至关重要。为解决该联合优化问题，本文进一步在搭载无人机的移动边缘计算系统中构建了基于深度强化学习（deep reinforcement learning, DRL）的框架，该框架结合双重深度Q网络（Double Deep Q-learning, D-DQN）与凸优化（convex optimization）技术，以在线方式联合求解地面终端间近似最优的无人机三维轨迹与任务卸载策略。借助所提框架，搭载无人机的移动边缘计算系统可快速完成各地面终端的计算任务，有效缩减无人机任务时长并降低推进能耗。数值仿真结果表明，基于深度强化学习的框架性能优于各类基准方案：以场景1为例，相较于传统深度Q网络（DQN）方案，系统吞吐量从64.24Kb提升至72.99Kb，无人机-地面终端链路的平均数据传输速率从4.72Kbps提升至20.75Kbps，无人机平均能效从0.32bits/J提升至0.36bits/J；其性能与现有旅行商问题（Travelling Salesman Problem, TSP）方案近乎持平，使得能效型边缘计算在搭载无人机的移动边缘计算系统中真正具备落地可行性。