Training Deep Spiking Neural Networks Using Backpropagation

Spiking neural networks promise something that conventional deep networks struggle to deliver in real-time and embedded settings: lower latency and higher energy efficiency through event-driven, data-dependent computation. The idea is elegant. Instead of processing everything all the time, spiking neural networks, or SNNs, only do work when something happens, specifically when a spike arrives. That approach maps naturally onto neuromorphic hardware and onto event-based sensors that output data in the same way. The potential here is real. However, that promise runs straight into a single, fundamental obstacle. Spikes are discrete, all-or-nothing events. A neuron either fires or it doesn't. Error backpropagation, which is the training method behind essentially every successful deep network, requires smooth, differentiable signals. This allows computation of each parameter's contribution to an error and how to adjust it in the right direction. The discontinuities from spikes break that calculation. You can't take a gradient through a threshold crossing. Lee, Delbruck, and Pfeiffer describe this as the core technical reason why deep, supervised training of SNNs has been so challenging to achieve. There's another complication they refer to as the dead neuron problem. To prevent runaway spiking activity, neurons need large firing thresholds. But large thresholds mean many neurons never fire at all during training. This results in them not contributing to the gradient and consequently not getting updated. They're essentially dead weight. Before this paper, researchers addressed these obstacles mainly in two ways, both of which come with real costs. The first approach is indirect training: you train a conventional artificial neural network in the usual manner and then convert its weights into a spiking network afterward. This method works on static benchmarks; for example, Diehl and colleagues reached 98.64 percent on the permutation-invariant MNIST through conversion, while Hunsberger and Eliasmith achieved 98.6 percent on the same task. However, conversion loses the spike-timing statistics. The converted network no longer captures how spikes are distributed over time, only their average rates. When you apply conversion to truly event-based data, like that from a dynamic vision sensor, accuracy drops sharply. Neil and Liu reported that an artificial neural network achieved 98.3 percent on a dataset derived from a dynamic vision sensor, but the converted spiking neural network fell to 95.72 percent on the same task. The second approach uses rate coding: neurons are modeled by their mean firing rates over time, which are continuous and differentiable, making gradient methods applicable. However, rate coding is slow and expensive. To obtain accurate rate estimates, you need to integrate over time. This means longer windows, more data, and more computation, which is precisely the opposite of what you want from an event-driven system. Lee, Delbruck, and Pfeiffer's insight is to sidestep both approaches entirely and train SNNs directly on spike signals. The trick is to treat the membrane potential—the continuous voltage that builds up inside a neuron before it fires—as the differentiable signal for learning, and to treat the abrupt jump at the moment of spiking as noise. The discontinuity doesn't disappear, but it's recast as something the optimizer can tolerate rather than something that disrupts it. The neurons in their networks are leaky integrate-and-fire units, or LIF neurons. Each incoming spike adds a decaying contribution to a neuron's membrane potential, which leaks away over a membrane time constant. When the potential crosses a threshold, the neuron fires, and the potential resets. For training, the authors compute low-pass filtered versions of these membrane contributions—smooth, continuous signals—and use those for stochastic gradient descent. The spike jumps at threshold crossings are skipped when computing gradients. The argument is that these discontinuities act like injected noise and do not harm optimization in practice. In addition, they incorporate winner-takes-all circuits, abbreviated as WTA. Groups of neurons are connected by lateral inhibitory links so that when one fires, it suppresses the others in its group. This arrangement promotes competition and sparsity. Lateral inhibition enters the differentiable transfer function as a subtractive term. The authors found that an inhibition strength of approximately negative 0.4 worked best in hidden layers, and negative 1.0 in the output layer. For gradient computation, they keep only first-order effects of these inhibitory connections. Deep networks also require careful initialization and gradient management. Lee and colleagues introduced an error-normalization scheme designed to keep the expected squared sum of backpropagated errors approximately constant across layers. The design goal is to maintain that quantity at one, ensuring that gradients do not explode or vanish as they propagate through multiple spiking layers. Weight initialization is drawn from uniform distributions scaled by a multiplicative constant alpha, set between three and ten depending on the architecture. These details may not sound glamorous, but they are what makes deep SNN training stable. Now for the results, and this is where the method demonstrates its value. On MNIST, the authors converted each handwritten digit image into a stream of Poisson spike events, with pixel intensity determining the event rate. A fully connected SNN with one hidden layer of one hundred units reached 98.64 percent accuracy on the permutation-invariant version of the task. In this version, inputs are randomized so the network cannot exploit spatial structure. With two hidden layers and the ADAM optimizer, the best results rose to about 98.77 to 98.88 percent over training epochs. These numbers match the best previous results for SNNs and are comparable to conventional artificial neural networks on the same setup. When spatial structure is allowed, and the authors employ a spiking convolutional architecture with data augmentation, accuracy on MNIST reaches 99.31 percent, which is competitive with top non-spiking convolutional neural network results. So, on artificial spike inputs derived from images, the method performs well. However, the results on N-MNIST are where things become genuinely striking. N-MNIST is a neuromorphic version of the handwritten digit dataset, recorded with a real dynamic vision sensor. A dynamic vision sensor is an event-based camera that outputs events whenever pixel intensity changes: an on-event for an increase and an off-event for a decrease, rather than capturing full frames at fixed intervals. To record N-MNIST, a dynamic vision sensor was moved in a three-phase saccade pattern in front of projected digit images, producing three hundred millisecond event streams per sample. The input to the network was thirty-four by thirty-four pixels with two channels for on and off events. On this dataset, a fully connected SNN with eight hundred hidden units achieved 98.66 percent accuracy, resulting in an error rate of 1.34 percent. The best previous spiking network on this task, a convolutional neural network from Neil and Liu, had an error rate of 4.28 percent. Lee and colleagues' method reduces that error by more than a factor of three. Compared to a conventional convolutional neural network trained and tested on the same N-MNIST data, which reached 98.3 percent accuracy with a 1.7 percent error rate, the SNN is more accurate, despite being fully connected rather than convolutional in architecture. The efficiency comparison is also crucial. To match the SNN's performance, the conventional artificial neural network had to accumulate events across ten consecutive thirty-millisecond frames, resulting in a three hundred millisecond window. This increased its computational cost by roughly 4.8 times compared to the SNN processing the same event stream directly. That reduction in operations by a factor of five is not just theoretical; it's what the two systems actually accomplish on the same task with the same data. The authors also note something revealing about how the SNN's accuracy evolves over time. As each saccade completes, accuracy jumps sharply, and each subsequent saccade roughly halves the error rate. This indicates that the network is performing useful inference before the full input arrives—something a frame-based system fundamentally cannot do. The broader implication here is hardware-facing. In a neuromorphic processor, where computational costs arise only when a spike occurs, these operational savings translate directly into energy savings. Lee, Delbruck, and Pfeiffer reference work by Merolla and colleagues on neuromorphic chips as the direction this training method aims for. The proposed pipeline consists of training an SNN directly on spike signals using the described method, deploying it on hardware that operates on events, and continuously reaping the efficiency gains. Open challenges remain. The authors clearly state that parallelizing event-based updates on conventional graphics processing unit hardware is challenging and that applying this training method directly on neuromorphic chips would necessitate measuring spike statistics at runtime, a capability that current platforms do not easily support. Additionally, MNIST and N-MNIST are relatively small benchmarks, making scaling to larger, more complex tasks an open question. However, the conceptual shift represents the lasting contribution. The spike was the obstacle—the element that disrupted backpropagation and pushed researchers into indirect workarounds. Lee, Delbruck, and Pfeiffer demonstrated that if you treat the membrane potential as the continuous signal and the spike jump as noise, the obstacle dissolves. You can train through the spike directly. This reframing is what opened the door. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.