Data from: Neural modularity helps organisms evolve to learn new skills without forgetting old

A long-standing goal in artificial intelligence is creating agents that can learn a variety of different skills for different problems. In the artificial intelligence subfield of neural networks, a barrier to that goal is that when agents learn a new skill they typically do so by losing previously acquired skills, a problem called catastrophic forgetting. That occurs because, to learn the new task, neural learning algorithms change connections that encode previously acquired skills. How networks are organized critically affects their learning dynamics. In this paper, we test whether catastrophic forgetting can be reduced by evolving modular neural networks. Modularity intuitively should reduce learning interference between tasks by separating functionality into physically distinct modules in which learning can be selectively turned on or off. Modularity can further improve learning by having a reinforcement learning module separate from sensory processing modules, allowing learning to happen only in response to a positive or negative reward. In this paper, learning takes place via neuromodulation, which allows agents to selectively change the rate of learning for each neural connection based on environmental stimuli (e.g. to alter learning in specific locations based on the task at hand). To produce modularity, we evolve neural networks with a cost for neural connections. We show that this connection cost technique causes modularity, confirming a previous result, and that such sparsely connected, modular networks have higher overall performance because they learn new skills faster while retaining old skills more and because they have a separate reinforcement learning module. Our results suggest (1) that encouraging modularity in neural networks may help us overcome the long-standing barrier of networks that cannot learn new skills without forgetting old ones, and (2) that one benefit of the modularity ubiquitous in the brains of natural animals might be to alleviate the problem of catastrophic forgetting.人工智能领域的长期核心目标之一，是构建能够针对不同任务习得多样化技能的AI智能体（AI Agent）。在神经网络这一人工智能子领域中，实现该目标的核心阻碍在于：智能体在习得新技能时，往往会丢失此前已掌握的技能，这一问题被称为灾难性遗忘（catastrophic forgetting）。其根源在于，为了学习新任务，神经学习算法会修改用于编码既往习得技能的神经连接。神经网络的架构组织方式，对其学习动态有着关键性影响。 本文中，我们将测试通过演化模块化神经网络（modular neural networks）是否能够缓解灾难性遗忘问题。从直觉层面而言，模块化设计可将功能拆解为物理独立的模块，并可选择性地启用或禁用各模块的学习进程，从而降低不同任务间的学习干扰。此外，若将强化学习模块与感官处理模块相分离，还可进一步优化学习流程，使得学习仅在接收到正向或负向奖励时才触发。本文中，学习过程通过神经调制（neuromodulation）实现，该机制允许智能体基于环境刺激，选择性调整每条神经连接的学习速率（例如，可根据当前任务在特定神经元位置调整学习行为）。 为了催生模块化结构，我们对神经网络施加了神经连接成本约束。我们证实，该连接成本方法可有效催生模块化神经网络结构，验证了此前的相关研究结论；同时，这类稀疏连接的模块化神经网络整体性能更优：一方面，它们能够更快速地习得新技能，同时更好地保留既往掌握的技能；另一方面，这类网络具备独立的强化学习模块。 我们的研究结果表明：其一，在神经网络设计中鼓励模块化结构，或可帮助我们突破长期存在的核心瓶颈——即神经网络在学习新技能时必然会遗忘旧技能；其二，自然动物大脑中普遍存在的模块化结构，其潜在益处之一或许正是缓解灾难性遗忘问题。