Data from: Neural modularity helps organisms evolve to learn new skills without forgetting old

A long-standing goal in artificial intelligence is creating agents that
can learn a variety of different skills for different problems. In the
artificial intelligence subfield of neural networks, a barrier to that
goal is that when agents learn a new skill they typically do so by losing
previously acquired skills, a problem called catastrophic forgetting. That
occurs because, to learn the new task, neural learning algorithms change
connections that encode previously acquired skills. How networks are
organized critically affects their learning dynamics. In this paper, we
test whether catastrophic forgetting can be reduced by evolving modular
neural networks. Modularity intuitively should reduce learning
interference between tasks by separating functionality into physically
distinct modules in which learning can be selectively turned on or off.
Modularity can further improve learning by having a reinforcement learning
module separate from sensory processing modules, allowing learning to
happen only in response to a positive or negative reward. In this paper,
learning takes place via neuromodulation, which allows agents to
selectively change the rate of learning for each neural connection based
on environmental stimuli (e.g. to alter learning in specific locations
based on the task at hand). To produce modularity, we evolve neural
networks with a cost for neural connections. We show that this connection
cost technique causes modularity, confirming a previous result, and that
such sparsely connected, modular networks have higher overall performance
because they learn new skills faster while retaining old skills more and
because they have a separate reinforcement learning module. Our results
suggest (1) that encouraging modularity in neural networks may help us
overcome the long-standing barrier of networks that cannot learn new
skills without forgetting old ones, and (2) that one benefit of the
modularity ubiquitous in the brains of natural animals might be to
alleviate the problem of catastrophic forgetting.

人工智能领域的一个长期目标是构建能够针对不同问题学习多种技能的智能体（AI Agent）。在人工智能的子领域神经网络中，实现这一目标的一大障碍在于：智能体学习新技能时，往往会丢失先前习得的技能——这一问题被称为灾难性遗忘（catastrophic forgetting）。其原因在于，为了学习新任务，神经学习算法会修改编码先前习得技能的连接。网络的组织方式对其学习动态（learning dynamics）有着关键影响。本文测试了通过演化模块化神经网络（modular neural networks）是否能减少灾难性遗忘。从直觉上看，模块化通过将功能划分为物理上独立的模块（可选择性开启或关闭模块内的学习），应能减少任务间的学习干扰（learning interference）。此外，模块化可通过将强化学习（reinforcement learning）模块与感官处理模块分离，使学习仅在正负奖励刺激下发生，从而进一步提升学习效果。本文中，学习通过神经调节（neuromodulation）实现，该机制使智能体能够根据环境刺激选择性调整每个神经连接的学习速率（例如，根据当前任务改变特定区域的学习状态）。为了生成模块化结构，我们通过为神经连接设置成本来演化神经网络。研究发现，这种连接成本技术可催生模块化结构（验证了先前的研究结果）；且此类稀疏连接的模块化网络具有更优的整体性能——原因在于它们学习新技能更快、保留旧技能更好，且拥有独立的强化学习模块。我们的结果表明：（1）在神经网络中促进模块化可能有助于突破“学习新技能必遗忘旧技能”这一长期存在的障碍；（2）自然动物大脑中普遍存在的模块化结构，其优势之一或许是缓解灾难性遗忘问题。