22

I'm working with Python 2.7 on Windows 8/XP.

I have a program A that runs another program B using the following code:

p = Popen(["B"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
return

B runs a batch script C. C is a long running script and I want B to exit even though C has not finished. I have done it using the following code (in B):

p = Popen(["C"])
return

When I run B, it works as expected. When I run A however, I expected it to exit when B exits. But A waits until C exits even though B has already exitted. Any ideas on what's happening and what possible solutions could be?

Unfortunately, the obvious solution of changing A to look like B is not an option.

Here is a functional sample code to illustrate this issue: https://www.dropbox.com/s/cbplwjpmydogvu2/popen.zip?dl=1

The zip file consists of the following files with the following contents:

A.py

from subprocess import PIPE, Popen
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("\r\n\r\nA: I'll wait for B\r\n")

p = Popen(["C:\\Python27\\python.exe", "B.py"], stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

log("A: Done.\r\n")
sys.exit(0)

B.py

from subprocess import Popen, PIPE
import sys

def log(line):
    with open("log.txt", "a") as logfile:
        logfile.write(line)

log("B: launching C\r\n")

p = Popen(["C.bat"])

log("B: Not waiting for C at all. bye!\r\n")
sys.exit(0)

C.bat

@echo off
echo C: Start long running task : %time% >>  "log.txt"
ping -n 10 127.0.0.1>nul
echo C: Stop long running task : %time% >>  "log.txt"

Any input is much appreciated.

6
  • If I get this right, you have program Am which runs program B which runs program C. Program A also runs program C. Is this correct? Commented Nov 6, 2012 at 5:27
  • No, program A doesn't run program C directly. It would be great if you took a look at attached example. Thanks.
    – khattam
    Commented Nov 6, 2012 at 5:32
  • So the second Popen is from program B? Commented Nov 6, 2012 at 5:35
  • Yes. A only runs B and has nothing to do with C directly.
    – khattam
    Commented Nov 6, 2012 at 15:45
  • Does supplying close_fds=True to the second Popen() call help? I'm guessing that C inherits the stdout/stderr pipes from A and thus A waits until C closes them.
    – Simon
    Commented Nov 6, 2012 at 16:48

3 Answers 3

31

You could provide start_new_session analog for the C subprocess:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

[1]: Process Creation Flags for CreateProcess()

9
  • 1
    This only works for me on windows if I had close_fds=True to the Windows kwargs line: kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True) (see this answer)
    – jtpereyda
    Commented Feb 28, 2017 at 3:33
  • 1
    @jtpereyda have you noticed that it says that you cannot use close_fds=True on Windows if you redirect any of stdin,stdout, stderr?
    – jfs
    Commented Feb 28, 2017 at 8:14
  • 2
    Combining DETACHED_PROCESS and CREATE_NEW_PROCESS_GROUP makes no immediate sense. Every process is in a group in Windows, even if it's just the group for the winlogon.exe or wininit.exe session, but the only API that uses process groups is GenerateConsoleCtrlEvent for sending Ctrl+Break (or Ctrl+C if the process manually enables it, since creating a new group initially disables it) to a group of processes that are attached to one's current console. Injecting the control thread is coordinated between the console instance (conhost.exe) and the Windows session server (csrss.exe).
    – Eryk Sun
    Commented Jul 18, 2017 at 18:28
  • 1
    The equivalent of a Unix process tree in Windows is a Job object. The process should be started suspended (creation flag CREATE_SUSPENDED) to ensure it's added to the job before it can spawn other processes that leak out of the job. This works best in Windows 8+ since it supports nested jobs, whereas older versions only allow a single job per process.
    – Eryk Sun
    Commented Jul 18, 2017 at 18:36
  • 4
    Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 Commented Jan 6, 2020 at 13:02
0

An alternative would be to start C as fully forked-off process in a separate process tree, e.g. via the start command of cmd.exe:

import subprocess
subprocess.Popen(["cmd.exe", "/C", "start notepad"])

Obviously, since it is completely independent, you cannot communicate with it. But you can use psutil to retrieve it's PID to at least monitor or close it, if necessary:

import psutil
for process in psutil.process_iter():
    if process.name() == 'notepad.exe':
        print(process.pid)
-1

Here is a code snippet adapted from Sebastian's answer and this answer:

#!/usr/bin/env python
import os
import sys
import platform
from subprocess import Popen, PIPE

# set system/version dependent "start_new_session" analogs
kwargs = {}
if platform.system() == 'Windows':
    # from msdn [1]
    CREATE_NEW_PROCESS_GROUP = 0x00000200  # note: could get it from subprocess
    DETACHED_PROCESS = 0x00000008          # 0x8 | 0x200 == 0x208
    kwargs.update(creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP, close_fds=True)  
elif sys.version_info < (3, 2):  # assume posix
    kwargs.update(preexec_fn=os.setsid)
else:  # Python 3.2+ and Unix
    kwargs.update(start_new_session=True)

p = Popen(["C"], stdin=PIPE, stdout=PIPE, stderr=PIPE, **kwargs)
assert not p.poll()

I've only tested it personally on Windows.

1
  • 3
    Note that starting from Python 3.7, on Windows it's no longer necessary to set close_fds=True - that is now the default and works together with redirecting stdin/stdout/stderr. Details in bug tracker here: bugs.python.org/issue19764 Commented Jan 6, 2020 at 13:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.