How to close file of FileHandler opened in Python process

Question

I'm using a logger inside Python process (multiprocessing.process) to log. I declare a FileHandler to log to disk, once the job is done I try to close the underlying file but can't achieve that. My real problem is that I spawn a lot of process resulting in IOError: [Errno 24] Too many open files

I reproduce the error with this snippet (test_process.py):

import logging
import multiprocessing

class TestProcess(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)
        self.logger = logging.getLogger('test')
        self.logger.setLevel(logging.INFO)
        self.handler = logging.FileHandler('test.log')
        self.handler.setLevel(logging.INFO)
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        self.handler.setFormatter(formatter)
        self.logger.addHandler(self.handler)

    def run(self):
        self.logger.info('hello')
        self.logger.removeHandler(self.handler)
        self.handler.close()

and when I run this:

from test_process import TestProcess
p=TestProcess()
p.start()
p.join()

You can check that there is file descriptor alive for the file test.log.

Any hint to overcome this behaviour?

Answer 1

I solved this issue adding the following to the class:

def __del__(self):
    self.logger.removeHandler(self.handler)
    self.handler.close()

Answer 2

That's because multiprocess uses forks, so anything in __init__ is run in the parent process, including your log file creation.

When you call start, the process forks and the file handle is shared between the two processes.

You should setup your logging in the run function. Then you don't have to worry about that.