Are neurons instantly feed forward when input arrives?

Question

Let's say I have a neural network with 5 layers, including the input and output layer. Each layer has 5 nodes. Assume the layers are fully connected, but the 3rd node in the 2nd layer is connected to the 5th node in the 4th layer. All these numbers are chosen at random for the example.

When is the 5th node in the 4th layer fed forward?

Let's go through it step by step. The first layer is normally fed forward to the second. The second layer is normally fed forward to the third, but the 3rd node is also fed forward to the 5th node of the 4th layer.

At this point, is the 5th node in the 4th layer now fed forward, or is it fed forward when the 3rd layer is done being fed forward? The 1st method would mean that the node would get fed forward 2 times and my concern is, if the output is still valid.

Furthermore, it would also come to 2 asynchronous outputs and how would these be interpreted?

Because in the brain, I heard, the neurons are fired when an impulse arrives so this would equal the 1st method.

Answer 1

It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in turn introduce new computations to the neural network.

This is your network graph if I got it right:

The calculation in a neural network without refractory periods and resting potentials without time, would instantaneously modify the weight of your node4 layer 5 (n4-L5) if there is input in column 3:

Additional inputs on other columns would just add up unless you have some other explicit computation on any layer.

If you wanted to emulate a Neuron, each node would need to have a resting potential, that is: a level above which it will fire ( in the above example zero), and a refractory period: a time before it would fire again, as well as a clock to keep it in sync, this would be a crude realtime fascimile:

http://codepen.io/k3no/pen/WRQbYV

A common alternative is to use sequential phases or steps.

Reference/source

Artificial Intelligence: A Modern Approach, by S. Russell and P. Norvig. Deals with a general step approach to A.I.

Gateway to Memory, by Mark A. Gluck and Catherine E. Myers Presents a great and readable introduction to modeling neural networks.

I published the little neural network model in a medium article : Memory and the machine, relevant sources are there.