When we set a batch-size, after each sample of batch passed we take the gradient but wait until last sample of batch to passed and then propagate the sum of gradient of them through the network? Am I correct or not?
If it isn't and we propagate after each pass of sample so what is the benefits of batch? Please, someone give me an explanation.
When training neural networks, backpropagation requires the computation of many gradients and it can be computationnally heavy. In order to minimize that load, the weights are only updated (i.e. the backpropagation is done) after a certain amount of samples, that's called batch (or mini-batch, if number of samples is low) gradient descent.
So, the loss function is of course computed after each training example and summed up for all the samples in the batch, then backpropagation is applied on the overall loss of the batch.
You can also update the weights after each sample, this is called stochastic gradient descent
