Bio-inspired Energy Optimizations to Synchronous Spiking Neural Network Architecture for Reinforcement Learning at Edge Application

Abstract

Reinforcement learning (RL) offers a reward-based model through trial and error within natural contexts, presenting an attractive approach to machine learning due to its simplicity, effectiveness, and similarity to human and animal learning processes. Particularly promising for neuromorphic implementation in autonomous agents' control, RL faces challenges in digital AI accelerators in micro-to-milli-Watt edge applications, where complex features inherited from traditional computing, such as floating-point arithmetic, increase cost and power consumption. Such approaches often struggle to efficiently accommodate biologically inspired models, where learning capacity and accuracy are customized for the task instead of relying on costly upfront hardware resources.

Our research within Energy-Aware Artificial Learning Group targets bio-inspired digital RL hardware model employing simplified integer arithmetic to enhance cost and energy efficiency. The model comprises a 16-node spiking neural network (SNN) template from literature where integer synapse weights adjust through reward and non-reward actions. First architectural optimizations delivered by our group highlighted potential significant energy savings compared to the literature by adopting small integer resolution and simplifying the implementation of Long Term Potentiation (LTP) and Long Term Depression (LTD) operations in Spike-Timing-Dependent-Plasticity (STDP) to track causality over only a few consecutive clock ticks. The implementation focused on a core RL network and required an external processor support for real-time autonomous execution.

This work introduces further bio-inspired optimization to the original RL architecture, providing substantial energy consumption benefits, while at the same time meeting the requirement for real-time autonomous processing with higher accuracy in context-dependent tasks. Synthesis, simulation, and functional validation of real-world implementation on dual-supply Intel MAX10 Field-Programmable Gate Array (FPGA) reveal about an order of magnitude reduction in average power dissipation and even larger benefits in energy consumption compared to state-of-the-art solutions, illustrating the potential of this SNN edge architectural approach.

Index Terms- Reinforcement learning (RL), spiking neural network (SNN), neuromorphic hardware, digital system design, low-power, low-cost, low-energy, leaky integrate and fire (LIF) model, context-dependent task, brain-inspired computing, FPGAs.