When training a neural network with back propagation, I have often seen that data is processed in batches. So instead of computing and updating the gradient for each training sample, the average gradient is calculated over multiple samples, this is used for the update.
What is the reason for this? Is it because it is faster to train as you update the weights less frequently? Or is it because training over multiple samples avoids overfitting to the individual samples? If the latter is try, then why not train over all the samples at once, rather than dividing it into batches at all?
Thank!
