Data_Sheet_1_Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning.docx

Artificial neural networks (ANNs) have been successfully trained to perform a wide range of sensory-motor behaviors. In contrast, the performance of spiking neuronal network (SNN) models trained to perform similar behaviors remains relatively suboptimal. In this work, we aimed to push the field of SNNs forward by exploring the potential of different learning mechanisms to achieve optimal performance. We trained SNNs to solve the CartPole reinforcement learning (RL) control problem using two learning mechanisms operating at different timescales: (1) spike-timing-dependent reinforcement learning (STDP-RL) and (2) evolutionary strategy (EVOL). Though the role of STDP-RL in biological systems is well established, several other mechanisms, though not fully understood, work in concert during learning in vivo. Recreating accurate models that capture the interaction of STDP-RL with these diverse learning mechanisms is extremely difficult. EVOL is an alternative method and has been successfully used in many studies to fit model neural responsiveness to electrophysiological recordings and, in some cases, for classification problems. One advantage of EVOL is that it may not need to capture all interacting components of synaptic plasticity and thus provides a better alternative to STDP-RL. Here, we compared the performance of each algorithm after training, which revealed EVOL as a powerful method for training SNNs to perform sensory-motor behaviors. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

人工神经网络（Artificial Neural Networks, ANNs）已被成功训练以完成广泛的感知运动行为。与之形成鲜明对比的是，经训练以完成同类任务的脉冲神经网络（Spiking Neural Network, SNN）模型的性能仍相对欠佳。本研究旨在通过探索不同学习机制的潜力以实现最优性能，从而推动脉冲神经网络领域的发展。我们采用两种运行于不同时间尺度的学习机制，训练脉冲神经网络以解决CartPole强化学习（Reinforcement Learning, RL）控制问题：其一为脉冲时序依赖强化学习（Spike-Timing-Dependent Reinforcement Learning, STDP-RL），其二为进化策略（Evolutionary Strategy, EVOL）。尽管脉冲时序依赖强化学习在生物系统中的作用已得到充分证实，但在体内学习过程中，尚有多种尚未被完全阐明的其他机制协同发挥作用。构建能够精准捕捉脉冲时序依赖强化学习与这些多样化学习机制间相互作用的精确模型，极具挑战性。进化策略作为一种替代方法，已在多项研究中得到成功应用：既可用于使模型的神经响应匹配电生理记录结果，在部分场景中也可用于解决分类任务。进化策略的一大优势在于，其无需捕捉突触可塑性的所有相互作用组分，因此相较脉冲时序依赖强化学习是更优的替代方案。本研究对两种算法训练后的性能进行了对比，结果表明进化策略是训练脉冲神经网络完成感知运动行为的高效方法。本研究的建模工作为强化学习领域的脉冲神经网络应用开辟了新方向，同时也可为旨在阐明神经元回路中多时间尺度学习机制与动态特性的神经生物学家提供研究测试平台。