本文为英国格拉斯哥大学（作者：Valente Klaine）的博士论文，共199页。

下一代移动网络5G及后5G时代必须克服目前的网络限制，并持续提高网络性能。展望未来移动网络的一些要求是：解决覆盖率、容量和通信量所需的大规模增长；为最终用户提供更好的服务质量和体验；支持超高数据速率和可靠性；确保低至1毫秒的延迟等。因此，为了使未来的网络能够满足所有这些严格的要求，一个很有前途的概念出现了：自组织网络（SON）。SON包括使移动网络更具适应性和自治性，并根据使用情况分为三个主要分支，即：自我配置、自我优化和自我修复。SON是一个非常有前途和广阔的概念，为了实现它，需要在移动网络中嵌入更多的智能。因此，一个可能的解决方案是利用机器学习（ML）算法。ML有许多分支，如有监督、无监督和强化学习（RL），它们都可以用于不同的用例。

本论文的目的是探讨不同RL技术在SON中的应用，更具体地说是在自优化用例中。首先，分析、优化了未来异构网络中用户-小区关联的用例。这个场景不仅考虑了无线接入网（RAN）的限制，而且还考虑了回传。在此基础上，提出了一种基于RL的分布式解决方案，并与现有的方法进行了比较。结果表明，所提出的RL算法性能优于现有的RL算法，能够获得更好的用户满意度，同时最大限度地减少中断用户数。

本论文的另一个目标是评估无人机（UAV）应用对蜂窝网络的优化。可以设想，无人机可以在不同的实例中使用，并与RL算法相结合，根据网络约束确定其在空间中的最佳3D位置。因此，分析了两种不同的移动网络场景，一种是紧急情况，另一种是pop-up网络。紧急情况设想认为，一场重大自然灾害摧毁了大部分地面网络基础设施，目标是向尽可能多的使用无人机作为接入点的用户提供通信覆盖。第二种情况模拟城市中发生的事件，由于地面网络拥塞，需要通过部署空中基站来增强网络容量。对于这两种情况，考虑了不同类型的RL算法，并分析了它们的复杂性和收敛性。在这两种情况下，无人机与RL相结合能够快速有效地解决网络问题。因此，由于RL能够从与环境的交互和以前的经验中学习，而不需要了解环境的动态，或者依赖以前收集的数据，因此RL被认为是一个很有希望的解决方案来实现SON。

The next generations of mobile networks 5G and beyond, must overcome current networks limitations as well as improve network performance. Some of the requirements envisioned for future mobile networks are: addressing the massive growth required in coverage, capacity and traffic; providing better quality of service and experience to end users; supporting ultra high data rates and reliability; ensuring latency as low as one millisecond, among others. Thus, in order for future networks to enable all of these stringent requirements, a promising concept has emerged, self organising networks (SONs). SONs consist of making mobile networks more adaptive and autonomous and are divided in three main branches, depending on their use-cases, namely: self-configuration, self-optimisation, and self-healing. SON is a very promising and broad concept, and in order to enable it, more intelligence needs to be embedded in the mobile network. As such, one possible solution is the utilisation of machine learning (ML) algorithms. ML has many branches, such as supervised, unsupervised and Reinforcement Learning (RL), and all can be used in different SON use-cases.

The objectives of this thesis are to explore different RL techniques in the context of SONs, more specifically in self-optimization use-cases. First, the use-case of user-cell association in future heterogeneous networks is analysed and optimised. This scenario considers not only Radio Access Network (RAN) constraints, but also in terms of the backhaul. Based on this, a distributed solution utilizing RL is proposed and compared with other state-of-the-art methods. Results show that the proposed RL algorithm outperforms current ones and is able to achieve better user satisfaction, while minimizing the number of users in outage. Another objective of this thesis is the evaluation of Unmanned Aerial vehicles (UAVs) to optimize cellular networks. It is envisioned that UAVs can be utilized in different SON use-cases and integrated with RL algorithms to determine their optimal 3D positions in space according to network constraints. As such, two different mobile network scenarios are analysed, one emergency and a pop-up network. The emergency scenario considers that a major natural disaster destroyed most of the ground network infrastructure and the goal is to provide coverage to the highest number of users possible using UAVs as access points. The second scenario simulates an event happening in a city and, because of the ground network congestion, network capacity needs to be enhanced by the deployment of aerial base stations. For both scenarios different types of RL algorithms are considered and their complexity and convergence are analysed. In both cases it is shown that UAVs coupled with RL are capable of solving network issues in an efficient and quick manner. Thus, due to its ability to learn from interaction with an environment and from previous experience, without knowing the dynamics of the environment, or relying on previously collected data, RL is considered as a promising solution to enable SON.

1.       引言

2. 项目背景与文献回顾

3. 基于RL的回传优化

4. 用于紧急移动网络的智能UAV

5. 用于pop-up网络的智能UAV

6. 结论与展望

附录A 比较Q学习与SARSA的示例

更多精彩文章请关注公众号：