Data_Sheet_1_Optical flow estimation from event-based cameras and spiking neural networks.PDF

Event-based cameras are raising interest within the computer vision community. These sensors operate with asynchronous pixels, emitting events, or “spikes”, when the luminance change at a given pixel since the last event surpasses a certain threshold. Thanks to their inherent qualities, such as their low power consumption, low latency, and high dynamic range, they seem particularly tailored to applications with challenging temporal constraints and safety requirements. Event-based sensors are an excellent fit for Spiking Neural Networks (SNNs), since the coupling of an asynchronous sensor with neuromorphic hardware can yield real-time systems with minimal power requirements. In this work, we seek to develop one such system, using both event sensor data from the DSEC dataset and spiking neural networks to estimate optical flow for driving scenarios. We propose a U-Net-like SNN which, after supervised training, is able to make dense optical flow estimations. To do so, we encourage both minimal norm for the error vector and minimal angle between ground-truth and predicted flow, training our model with back-propagation using a surrogate gradient. In addition, the use of 3d convolutions allows us to capture the dynamic nature of the data by increasing the temporal receptive fields. Upsampling after each decoding stage ensures that each decoder's output contributes to the final estimation. Thanks to separable convolutions, we have been able to develop a light model (when compared to competitors) that can nonetheless yield reasonably accurate optical flow estimates.

基于事件的相机正受到计算机视觉社区的日益关注。这类传感器采用异步像素工作机制，当单个像素自上一次产生事件以来的亮度变化超过特定阈值时，便会向外发出事件（或称“脉冲尖峰”）。得益于其低功耗、低延迟、高动态范围等固有特性，这类传感器尤其适配对时间约束严苛且有安全要求的应用场景。基于事件的传感器与脉冲神经网络（Spiking Neural Networks, SNN）适配性极佳，因为异步传感器与神经形态硬件结合后，可构建出功耗极低的实时系统。本研究旨在开发此类系统之一，借助DSEC数据集的事件传感器数据与脉冲神经网络，实现驾驶场景下的光流估计。我们提出了一种类U-Net结构的脉冲神经网络，经监督训练后可输出稠密光流估计结果。为此，我们同时优化误差向量的范数最小化与真实光流和预测光流间的夹角最小化目标，并通过代理梯度实现反向传播来训练模型。此外，通过使用三维卷积，我们可通过扩大时间感受野来捕捉数据的动态特性。每个解码阶段后均执行上采样操作，确保每个解码器的输出均可对最终估计结果做出贡献。得益于可分离卷积技术，我们得以构建出相较于同类竞品更为轻量的模型，且仍可输出精度可观的光流估计结果。