Relationship between batch size and the number of neurons in the input layer

Question

Regarding LSTM neural networks, I am unable to understand the relationship between batch size, the number of neurons in the input layer and the number of "variables" or "columns" in the input. (Assuming that there is a relationship and despite seeing examples to the contrary, I cannot understand why there is no relationship)

For the sake of clarity, I am going to use an example to formulate my query.

Let's assume that the dataset contains three columns of input and one column of output. So it will be something like input variable 1 input variable 2 input variable 3 output variable 1

From what I understand, the input layer of the LSTM network has to have 3 neurons corresponding to each of the input variables. It cannot be less or more. Even though I have seen examples like this answer (which appears to be very well described but unfortunately, I am not able to comprehend it.)

Now let us say that we have 50 rows of the above 4 columns.

That is essentially to me means that we have 50 samples. Now if the batch size is 5, then how many input neurons we have? Is the number of neurons in the input layer independent of the batch size? The way I understand batch size is that it is the number of samples that the neural network will see before updating its weights. So let's assume if we have only three neurons in the input layer, then we will be passing the first row of input variable, followed by the second row of input variable and repeating it until the fifth row before we update the weights. How is that going to be any different than just passing the fifth row?

Answer 1

In general I think you've understood both concepts. I'll try to address both of them in more detail.

Input layer size

In Neural Networks (NN), the size of the input layer is always equal to the number of variables (or features as we usually call them in Machine Learning) in your data.

Batch size

This refers to how many samples of data the network will see before updating its weights.

Normally, when performing gradient descent (i.e. the algorithm that trains the NN), the network would be input all of the samples in order to calculate the loss and update the weights. However, especially in Deep Neural Networks (DNNs) this is infeasible, as the size of the dataset is too big and even if the network had enough memory the computation of the gradients would be inefficient. What we do instead is pass a few samples (called a batch) at a time and train the model on them. Then another batch is passed, and so on. The number of samples we input to the model at each iteration is called the batch size.

Relationship between the two

In theory, if you have many features (in the thousands) you can't use a large enough batch size and the more features you have the less of a batch size you can use. However, in practice I have found the two to be practically unrelated! In Deep Neural Networks, the batch size is primarily governed by the size of the model. An easy (but not always accurate) way of estimating this size is through the number of trainable parameters in a model. More parameters, mean more memory for the model (and longer gradient computations), leading to a smaller batch size. In DNNs, that have multiple layers, only a fraction of the parameters are those of the input layer. Thus, the number of features plays a minor part in the selection of the batch size.

Example

In your example you say you have 50 rows and 4 columns, out of which 3 are the input variables.

The number of neurons in the input layer will be 3, to match the number of input variables (or features). This we cannot change, even if we wanted to!

The batch size is something we can change. Let's say we use a batch size of 5.

1st iteration:
In the first iteration, the NN will be given the first 5 rows of data. For each, the first three columns will be passed to the first layer of the network, which will calculate an output. After all 5 outputs are generated, they will be compared with the first 5 rows of the 4th column, to calculate a loss. Using this loss, the network will compute the gradients and perform the updates. Again, this loss corresponds to the first 5 samples, not just the 5th.

2nd iteration:
The rows 6-10 will be passed to the network, which will do the same procedure as in the first step.

10th iteration: The rows 45-50 will be passed to the network. This marks the point where the NN has seen every sample of data once. In the context of Machine Learning, we call this an epoch. At this point, the data are shuffled and the 1st iteration of the 2nd epoch begins.

Note:

In most frameworks the samples in each batch are processed in parallel by the network and not sequentially. This makes no difference to the result, I just wanted to point this out.

Answer 2

I do NOT agree with @Djib2011 on what he said "The number of neurons in the input layer will be 3, to match the number of input variables (or features). This we cannot change, even if we wanted to!" The number of neurons, also called units or nodes, is a totally free parameter which is specified up to you. It does not have to match the number of features. It determines the dimensions of the output of this layer though.

Check the example here https://keras.io/layers/core/, in which the first layer

model.add(Dense(32, input_shape=(16,)))

Here 32 is the number of neurons, and 16 is the number of feature. Another hidden layer

model.add(Dense(32))

where 32 is also the number of neurons, and in keras you do not need specify the dimension of input (output from previous layers) after first hidden layer (p.s. the input layer is not a real layer but a tensor).

Also please check this example https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/, in which the author mentioned:

We can piece it all together by adding each layer:

• The model expects rows of data with 8 variables (the input_dim=8 argument)
• The first hidden layer has 12 nodes and uses the relu activation function.
• The second hidden layer has 8 nodes and uses the relu activation function.
• The output layer has one node and uses the sigmoid activation function.

# define the keras model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))