Keras: release memory after finish training process

Question

I built an autoencoder model based on CNN structure using Keras, after finish the training process, my laptop has 64GB memory, but I noticed that at least 1/3 of the memory is still occupied, and the same thing for the GPU memory, too. I did not find out a good method to release the memory, I could only release the memory by closing the Anaconda Prompt command window and jupyter notebook. I am not sure if anyone has a good suggestion. Thanks!

Answer 1

Releasing RAM memory

For releasing the RAM memory, just do del Variables as suggested by @nuric in the comment.

Releasing GPU memory

This is a little bit trickier than releasing the RAM memory. Some people will suggest you the following code (Assuming you are using keras)

from keras import backend as K
K.clear_session()

However, the above code doesn't work for all people. (Even when you try del Models, it is still not going to work)

If the above method doesn't work for you, then try the following (You need to install the numba library first):

from numba import cuda
cuda.select_device(0)
cuda.close()

The reason behind it is: Tensorflow is just allocating memory to the GPU, while CUDA is responsible for managing the GPU memory.

If CUDA somehow refuses to release the GPU memory after you have cleared all the graph with K.clear_session(), then you can use the cuda library to have a direct control on CUDA to clear up GPU memory.

Answer 2

For clearing RAM memory, simply delete variables as suggested by Raven.

But unfortunately for GPU cuda.close() will throw errors for future steps involving GPU such as for model evaluation. A workaround for free GPU memory is to wrap up the model creation and training part in a function then use subprocess for the main work. When training is done, subprocess will be terminated and GPU memory will be free.

something like:

import multiprocessing

def create_model_and_train( ):
      .....
      .....

p = multiprocessing.Process(target=create_model_and_train)
p.start()
p.join()

Or you can create below function and call it before each run:

from keras.backend.tensorflow_backend import set_session
from keras.backend.tensorflow_backend import clear_session
from keras.backend.tensorflow_backend import get_session
import tensorflow
import gc

# Reset Keras Session
def reset_keras():
    sess = get_session()
    clear_session()
    sess.close()
    sess = get_session()

    try:
        del classifier # this is from global space - change this as you need
    except:
        pass

    print(gc.collect()) # if it does something you should see a number as output

    # use the same config as you used to create the session
    config = tensorflow.ConfigProto()
    config.gpu_options.per_process_gpu_memory_fraction = 1
    config.gpu_options.visible_device_list = "0"
    set_session(tensorflow.Session(config=config))