Towards Making a Cyborg: A Closed-Loop Reservoir-Neuro System

The human brain is a remarkable computing machine, i.e. vastly parallel, self-organizing, robust, and energy efficient. To gain a better understanding into how the brain works, a cyborg (cybernetic organism, a combination of machine and living tissue) is currently being made in an interdisciplinary effort, known as the Cyborg project. In this paper we describe how living cultures of neurons (biological neural networks) are successfully grown in-vitro over Micro-Electrode Arrays (MEAs), which allow them to be interfaced to a robotic body through electrical stimulation and neural recordings. Furthermore, we describe the bioand nano-technological procedures utilized for the culture of such dissociated neural networks and the interface software and hardware framework used for creating a closed-loop hybrid neuro-system. A Reservoir Computing (RC) approach is used to harness the computational power of the neuronal culture.


Introduction
Engineered products are generally constructed from a number of unique, heterogeneous components assembled in very precise ways according to elaborate blueprints made by their designers. In contrast, life applies a completely different engineering paradigm to construct complex, reliable and adaptive structures, e.g. biological neural networks, from a basic set of organizational principles. Considering the functionality of living systems, the translation of these principles into technology will enable broad areas of engineering to move towards frontiers not reachable by current methodology.
Our brains are able to achieve prodigious feats of computation with remarkable speed and efficiency, including navigation in complex environments, object recognition, decision making, and reasoning. Many of these tasks have not been adequately solved using algorithms running on our most powerful computers. When one examines the physical structure of biological neural networks, it is indeed very difficult to understand how such systems are able to grow, self-organize, and perform computation.
Self-organization and growth are properties that not only challenge the common top-down engineering paradigm, but also challenge the basic view of what an engineered system is and its functionality (behavior). The design of com-plex, reliable and adaptive structures requires to move towards bottom-up design principles. An approach for a design concept for such autonomous systems capable of developing complex heterogeneous morphologies is Morphogenetic Engineering (Doursat et al., 2012); a concept where structures emerge as a product of interaction between autonomous units. Whatever the motivation for engineering a system is, there exist some target purpose for the system. Herein the target is computation or information processing. As such, the self organizing structure is the architecture for an information processing device. This architecture is a nonstatic structure and thereby giving rise to non-static functionality, i.e, a two way coupling between dynamic structure and functionality -a system with a possibility of inducing perturbations to its own dynamics as a function of its system states, which enables state space trajectory changes and topological reconfigurations of the state space (Omholt, 2013) -A Dynamical Systems with Dynamical Structures ((DS) 2 ) (Spicher et al., 2004).
In the last few years, the development of techniques to record and stimulate extracellular potentials using Micro Electrode Array hardware has opened new possibilities for the embodiment of in-vitro living neurons with robotic bodies. Biological neural networks that develop when connected to a body can exploit sensory information to shape aspects of the neural network itself. In addition, connecting neuronal networks to the outside world through sensory perception might be important to understand cognitive functions in the wide sense (through embodiment).
The Cyborg project aims at developing a cyborg by allowing communication between a machine and in-vitro hierarchical neural network. Such an approach has advantages compared to in-vivo systems, as it allows growing tailored neural networks with target functionalities as well as controlling their regulatory structure. Major conceptual and methodological advances are expected in the following areas: puter architecture may decrease energy consumption. In systems with limited energy availability, i.e. drones and robots, principles from small plastic networks may allow complex computation, e.g. a dragonfly can intercept a moving midair target with a network of only 16 neurons (Gonzalez-Bellido et al., 2013).
• Biomedical: regenerative and translational neuroscience, modelling and understanding damage and repair processes applicable to trauma or neurodegenerative disease, and developing novel approaches in nanomedicine and personalized medicine. Being able to elucidate how biological neural networks self-organize and grow may provide a stepping stone towards novel neuro-rehabilitation techniques.
• Philosophical: basic issues around neuronal functions are investigated. In the long term, studying the mechanism of memory, learning, concept formation, and neuronal model building of the external world may lead to a better understanding of the emergence of consciousness.
This paper presents an infrastructure for a closed-loop neuro-system that provides the capability for recording and stimulation of an in-vitro neuronal culture. The biological neural network is exploited as a reservoir of dynamics in order to provide control of an embodied agent, i.e. a cyborg. Results of spontaneous recordings of pace-maker neural activity are given, together with responses under stimulation (electrical and chemical) that serve as proof-of-concept of neural training. Further, results of neuro-controlled agent in a closed-loop system are presented. The paper is laid out as follows: Section 2 provides background information and Section 3 outlines the experimental setup. In Section 4 the experimental results are presented together with analysis. Section 5 concludes and Section 6 explores future works.

Background Reservoir Computing
Artificial neural networks (ANNs) represent a class of computational models that take inspiration from biological neural networks (BNNs). In ANNs, artificial neurons (the computing elements) are typically arranged in layers, i.e. neurons in one layer are connected to neurons in the next layer and information flows in a feed-forward fashion. Artificial recurrent neural networks (RNNs) are a more plausible and realistic model of BNNs, where the connection topology of neurons has recurrencies, i.e. cycles. By allowing cycles, the RNN becomes a dynamic system with a self-sustained temporal activation and possesses memory of previous inputs, i.e. the activation state of the network is a function of previous activation states. Such property is known as "echo state" (Jaeger, 2003). Artificial RNNs are far more powerful than feed-forward ANNs but they are also far more difficult to train, because learning gradients dissipate over Figure 1: Schematic overview of reservoir computing system. The reservoir itself can be any medium with desired properties, e.g. a biological neural culture or a artificial recurrent neural network. Adapted from (Schrauwen et al., 2007).
time (making it difficult to learn long-range memory dependencies) and network dynamics can lead to bifurcations. It has recently been suggested that both artificial and biological RNNs may be considered as a high-dimensional medium of dynamics with the ability to represent information in a high-dimensional and discriminating space. As such, the RNN can be treated as an "untrained" reservoir of dynamics and only a single linear readout layer is therefore needed for training. Figure 1 shows a schematic representation of a reservoir computing system. The figure is adapted to the use of an in-vitro neuronal culture as reservoir. W represent the trainable weights. W bias , W res and W f eedback are used to influence the structure and dynamics of the emerging neural network. These three are signals that can be used to influence the development of the morphological structure and behavior. As such, it is a true (DS) 2 architecture. Different substrates have been shown to possess the necessary rich dynamics to act as reservoirs. In Jaeger (2003), use echo state networks and in Maass et al. (2002) liquid state machines are introduced. In Nikolić et al. (2006) the primary visual cortex of anesthetized cats was investigated. Fernando and Sojakka (2003) implemented a reservoir in a bucket of water. In Nichele and Gundersen (2017); Nichele and Molund (2017) the use of cellular automata as reservoir is presented. An optoelectronic reservoir is described in Larger et al. (2012) and carbon-nanotube materials are evolved into reservoirs in Dale et al. (2016). Mechanical systems have also been used as reservoirs (Hauser et al., 2011(Hauser et al., , 2012. Recently, dissociated neuronal cultures have been investigated as a reservoir of dynamics (Takahashi et al., 2016).
In order to harness the innate computational power of the biological reservoir, the challenge becomes finding transforms from the input to the reservoir and interpreting the resulting behavior.

Proof-of-principle of embodied bio-robotic studies
Several studies have reported the ability and potential of developing hybrid bio-robotic systems utilizing Micro-Electrode Array-interfaced in-vitro neural networks: (Warwick et al., 2011;Xydas et al., 2008;Warwick et al., 2010) re-embodied dissected neurons from the brains of embryonic rats through a robotic body. DeMarse et al. (2001); DeMarse and Dockendorf (2005)  Notwithstanding that the above results are preliminary, these studies clearly demonstrate that it is indeed possible to establish self-organizing hierarchical neural networks on MEAs, interface them with a computer, and study their spontaneous neural network activity as well as their responses to electrochemical modulation longitudinally invitro. However, these studies have yet to display explicit learning within the networks other than the adaptive behavior caused by the networks intrinsic plasticity. Yet, this reported "learning by habit" (Warwick et al., 2011) behavior is a vital and fundamental step in getting there.

Recording extracellular activity of neural networks
A particular property of neurons is their ability to set up a potential difference between the inside and the outside of their cell membrane. This potential difference is set up by the interplay of diffusion, ion channels and active ion pumps. This cross-membrane potential essentially prepares the neuron for a 'spike' of electrical activity; a rapid polarity shift (seen as a voltage spike) across the membrane. These spikes may be triggered via electrical or chemical synaptic input from upstream neurons, or by electrical or chemical manipulation of the extracellular environment. During the spike, this polarity change propagates along the neurons axon, a long tendril that can extend to close and far away neurons, causing similar polarity changes, and possibly spikes, in the downstream neurons.
It is this electrical property of neurons one can take advantage of in order to record and stimulate a network of neurons by use of an MEA. By growing a dissociated neural network on top of the MEA, it is possible to monitor the extracellular voltage fluctuations that occur in the network in relation to an on-board reference electrode (ground). In addition, one may also stimulate the network by injecting a current through one or several of the MEAs electrodes. This current causes a shift to the cross-membrane potentials of nearby neurons which, if the stimulation is sufficiently strong, may result in these neurons spiking.
Even though these networks communicate in a complex interplay between both neurotransmitters and electrical spikes, there is a high degree of correlation between the electrical and chemical signals in these neural networks. Thus, a necessary simplification of considering only the electrical signals when interacting with neural networks can be made at a small cost of information loss.

Embodied Intelligence
Does the brain control the way the body behaves or does the body shapes the functioning of the brain? The body/brain issue is a kind of chicken and egg problem (Funes and Pollack, 1999). The course of natural evolution shows a history of body, nervous system, and environment all evolving simultaneously in response to each other (Pfeifer and Bongard, 2006).
In humans, aspects of the human cognition are shaped by aspects of the body (beyond the brain). Intelligence and cognition include high level mental constructs (concepts and categories) and human performance on various cognitive tasks (reasoning and judgment). Among the aspects of the body that influence cognition are the motor system, the perceptual system, and the bodys interaction with the environment. In Bongard et al. (2006), a robot infers his own body morphology and creates compensatory behavior in case of failure of some body parts. This may be considered an early development of cognition. In Anetsberger and Bongard (2016) the symbol grounding problem in simulated robots is investigated, by association to physical perceptions. Embodying an in-vitro neural network with a body in a closed-loop system may thus enable testing hypotheses on behavior and cognition, as well as provide insight into the nature of intelligence.

Experimental Setup
In this section, the experimental setup for two sets of experiments is presented. The first part (section Neural interface: MEA2100 and In Vitro Neural Network Culture Setup) deals with setting up the neural interface and describes the techniques utilized in order to grow the target neural networks. The second part (sections Distributed infrastructure-Reservoir Computing Setup) describes how these networks have been implemented in an embodied closed-loop neurorobotic system. comprises of a tissue culture dish incorporating bidirectional electrodes allowing direct contact with the neuronal culture. The standard system utilizes 60-electrode MEAs with an internal reference electrode and titanium nitride recording electrodes (MultiChannelSystems, 2017b). The system enables acquisition and analysis of electrophysiological data from the biological networks, as well as enables electrical stimulation of the networks. Each electrode can detect the extracellular activity of nearby neurons and can stimulate activity as described in section Recording extracellular activity of neural networks.

In Vitro Neural Network Culture Setup
The neural networks used consisted of motoneurons (MNs) or dopaminergic (DA) neurons morphogenetically engineered from induced pluripotent stem cells (iPSCs) through controlled expression of patterning factors at specific time points over a period of 30 days and 16 days, respectively, allowing partial recapitulation of developmental processes. The MEA and a neural culture are shown in Figure 2. The resulting cells were confirmed as MNs or DA neurons by positive expression of approperiate molecular markers including neuronal marker beta-III tubulin (Tuj-1), and Islet-1 or tyrosine hydrohylase, respectively. A total of 100,000 neurons from each type were seeded separately, onto the centre of a poly-L-ornithine (PLO)/laminin coated MEAs, and allowed to further mature and were maintained for electrophysiolog-ical recordings and stimulation.

Distributed infrastructure
To enable embodiment of the neuronal cultures, we need a system which allows for real-time bidirectional communication between the culture and a robotic body. This has been achieved by designing a closed-loop system where the neural recordings serve as motor instructions to a simulated robotic body, while sensory data from the robot is sent back to the culture through appropriate stimulation. This sensory feedback is essential in order to enable the network to learn something about its environment. The implemented infrastructure essentially consists of a four node system as shown in Figure 3. The three main components for real-time operation include: • The Host server MEA: establishes an interface between the MEA2100 (MultiChannelSystems, 2017b) and the rest of the network. This server provides clients with realtime MEA data and also performs MEA stimulation requests.
• The Host server Robot: establishes the bi-directional interface to the robotic body in which the biological networks are embodied.
• The Host server Interpretation: the workhorse of this setup. This node is responsible for applying the necessary data transformation algorithms and training protocols between the MEA and the robot.
In addition to these, a fourth important component is the Storage database which logs all experimental data.  Warwick et al., 2010Warwick et al., , 2011, and/or by utilizing the network's intrinsic stimulation response (Novellino et al., 2007;Bakkum et al., 2007) and assigning these responses to motor behaviors. Here, we propose a novel approach using reservoir computing in order to take advantage of the neural network's complex non-linear dynamics. The reservoir computing system that has been designed, is shown in Figure 4. This conceptual model shows the neuronal culture acting as a reservoir interfaced with a computer acting as a bridge between the reservoir and a robot. In order to bridge the neural reservoir and the robot, the computer is responsible for processing and filtering electrical readouts from the neuronal reservoir, mapping them to actions to be performed by the robot, and serving as an output layer for the reservoir. Likewise, sensory data from the robot must be mapped into stimuli that can be applied to the neuronal reservoir, serving as the input layer for the reservoir. By using this setup, the behavior of the closed-loop system can be altered by changing the input and output layers, foregoing the need to directly influence the growth of the neuronal culture. In our implementation, a simple single-layer artificial feed-forward network has been chosen as output layer, while a simple transform between sensory data (e.g. the robot distance to wall in a wall avoidance task) and period of stimuli on an electrode serves as the input layer. The choice of a feed-forward ANN is motivated by ease of implementation, not by any similarity to the real neurons. This choice allows for using a non-linear filter during the proof-of-concept phase, while retaining the possibility of reducing the network to a linear classifier in the form of a perceptron with no other changes.

Embodiment of Reservoir Computing System
We have currently embodied the neuronal culture through a simulated body. Through providing the biological networks sensory feedback in the form of extracellular stimulation, as well as converting network activity into robotic motor Figure 4: The proposed architecture for a cyborg using neuronal cultures as a reservoir. The readout layer can be any linear single-layer trained network or a multi-layer network, in case of more complex or multiple tasks. behavior, we enable a closed-loop system inspired by the sensory-motor loop vital for animal perception and movement. A simulated robot, or virtual creature (DeMarse et al. (2001) coined the term Animat) provides an easy and safe setup for testing the distributed neuro-robotic system and allows for the environment to be simplified. The animat here, consists of a body with four eyes and two motors, allowing it to sense the environment in a cone and steer either left or right. The environment that the animat inhabits is a small box with no features other than the four enclosing walls. In this very simple environment the animat can be trained to perform simple tasks such as wall avoidance, or skirting the walls without getting too close or too far.

Agent Bot and Objective
As a proof-of-concept, the animat is subjected to tasks that are trivial to solve with existing computers. The chosen task is using sight in order to avoid collision with a wall, which is a fairly simple task where performance can be easily quantified. To gauge animat performance, the animat faces a series of trials where it is placed close to a wall at different angles and points are awarded for total distance between the agent and the wall with zero points offered for a collision. With this setup, multiple animats can be created and measured against each other to search for a configuration that best solves the task presented. This is possible because we define an animat not only as a neuronal culture, but by its input and output layer. By this logic two animats differ as long as either their input or output layer is different, even though they are both powered by the same neuronal culture. However, in order to reduce the search space however the output layer has been fixed as a linear function of the agents distance to the wall seeing as the system should be more than powerful enough to compensate for a suboptimal output layer. Much like the choice of filters, the training algorithm can easily be swapped. In the current system only a simple genetic algorithm has been utilized, again arbitrarily chosen due to ease of implementation.

Results and Analysis
This section presents the results for the two main experiments: I) recording of spontaneous activity and responses under stimulation, which constitutes a stepping stone towards the closed-loop system used in II) agent control.

Spontaneous Activity and Stimulation Responses
A biological neural network was developed to act as a dynamic reservoir for the artificial neural network. The development of the neural network was gradual with three discrete phases. After seeding, the culture showed the following development of activity: Spontaneous tonic firing occurred after 20 days in vitro (DIV), brief bursts after 28 DIV, pacemaker burst at 40 DIV (See Figures 6, 7 and 8 respectively). Pacemaker bursts were able to drive activity throughout the culture and had a single point of origin. The burst propagation from the pacemaker cluster spread to the majority of the cluster, with an increasing number of nodes responding as the culture matured. Pacemaker activity was therefore chosen as a target for stimulation, with timing and signal strength allowing pinpointing of pacemaker origin. Pacemaker bursts were originally disordered with no clear timing to predict their arrival. To test the learning potential of the culture, we attempted stimulation to alter the timing of the pacemaker.
A simple biphasic pulse of -/+ 500 mV was chosen as the stimulus, with the pacemaker cluster as the stimulation target. Timing was set at 30 second intervals, with responses to stimulation gathered from nodes downstream of the pacemaker. Stimulation occurred for 10 minutes daily. Stimulation altered the pacemaker timing during the first day, however the timing became disordered once stimulation was disconnected. However, after 5 sessions of stimulation, the pacemaker maintained the 30 second interval in the absence of stimulation, suggestive of long term potentiation (LPT).
This timing later became disorganized with bursts occurring at 1-2 minute intervals. The cause of this is unknown as the stimulation would sometimes restart the 30 second timing, but only for brief periods. It is possible that as the culture is still undergoing significant proliferation and development, the network is too dynamic to impact long term. Indicative of this is the duration of pacemaker bursts, with burst trains changing from 1 second duration during the second month to 2-3 minutes during the forth. Secondly, the stimulation is not synchronized with the network activity and is instead enforced without regard to network state. As computational modeling is established, this synchronization will be among the initial goals to improve stimulation protocols. Despite the disorganized timing, pacemaker bursts are still present after 200 DIV.
In addition to electrical stimulation, chemical stimulation of the DA neurons utilizing dopamine was attempted in three instances. Upon testing, the DA culture responded vigorously to chemical stimulation by direct dopamine addition to the culture medium. All together 3,5M of dopamine was added to the culture at 32 DIV. After stimulation the culture was supplied with fresh media. During the second and third stimulation, at 37 DIV and 50 DIV, respectively, 3M of dopamine was added to the culture medium. The effect of dopamine addition was primarily enhancement of the fir-   ing strength, bringing tonic firing from 40-50 mV to 80-100 mV depending on the electrode and experiment. Little to no effect was seen on the burst duration and timing.
The experiments herein provide a proof-of-concept of recording neural activity as well as stimulation protocols, which are a requisite for the following experiment.

Agent performance
In the current iteration the agent is not able to achieve sophisticated behavior, but quickly converges towards running in circles. Although the agent currently fails to display more complex and interesting behavior, the stated goal of achieving a closed-loop system, showing that it is fea-sible to control an agent in real time over network protocols, has been successfully achieved. A video of the realtime agent behavior powered by the neural culture using a reservoir-computing setting is available at the following link: https://youtu.be/NcF0Uc-YqF8. This represents a stepping stone for further investigation of complex adaptive behaviors of the embodied agent.

Conclusion
In this paper we have presented a novel closed-loop hybrid neuro-system which uses in-vitro neural networks as a reservoir of dynamics. The activity of biological neural networks has been recorded with and without stimulation, and a proofof-concept of training properties has been provided. In addition, the neuronal culture has been embodied to provide sensory information as stimulation to the biological neural network through an MEA. This work lays the foundation for further studies within neuroscience, computer science and cybernetics, in a long-term endeavor towards a better understanding of the brain "language" and "functions". Directions for future work are discussed in the following section.

Future Work
Structuring the biological neural networks.
Apart from arrays utilizing self-organizing hierarchical neural networks in standard MEAs, future work will also include the study of equivalent networks within microfluidics devices with embedded microarchitecture, developed in-house. These microchips contain multiple cell compartments (nodes) interconnected through microtunnels permissible only to neuronal axons.
The microfluidics microchips, made with polydimethylsiloxane (PDMS), are gas-permeable and biocompatible and thus enable the establishment of in-vitro biointerfaces sustainable at standard cell culture conditions (Whitesides, 2006;Halldorsson et al., 2015). Furthermore, by incorporating a microelectrode interface, the microchips become compatible with the M2100 system, allowing for electrophysiological recording of neural network activity as well as electrochemical stimulation (Pan et al., 2015). Thus, the PDMS microarchitecture enables selective electrical stimulation and recording of isolated nodes, while it also facilitates signal propagation to other nodes via the interconnecting axons.
Currently, a recently-developed multi-nodal microfluidic chip is being established. Pilot studies are focused on protocol optimization, especially in terms of scaling neuronal culture and imaging parameters. In the near future, electrodes will be integrated into the surface of this chip to finalize the first version of the interface.

Physical robot: Cyborg interactive robot
As part of creating a cyborg, the Cyborg project is currently developing an interactive robot which will in the future serve as the a body for embodying the biological networks. In conjunction with this, a Pioneer LX(MobileRobots, 2017b) navigation base, developed by Omron Adept MobileRobots(MobileRobots, 2017a), has been purchased. On top of this base, we are developing necessary hardware and software to meet this goal of an interactive physical cyborg.

Computational complexity of neuronal cultures reservoir
Being able to embody a neuronal culture through a reservoir computing paradigm is a stepping stone towards a better understanding of the computational capabilities and selforganizing dynamics of neurons. The developed closed-loop system will serve as multi-disciplinary platform to study and elucidate the underlying mechanisms of memory and learning in biological neural networks, as well as hypothesis on concepts formation and emergence through local interactions.