Is it possible for CMD.EXE to release a redirected output file before all sub-processes exit?

Question

I have a program that launches other instances of itself. Each instance redirects the standard output stream to a different file. I'm having an issue with the following chain of events:

1. A BAT file gets executed that contains the command my_program.exe > output_launcher.txt
2. my_program.exe launches more instances of itself using the commands my_program.exe > output_1.txt and my_program.exe > output_2.txt
3. The original instance (my_program.exe > output_launcher.txt) exits while the two launched instances (my_program.exe > output_1.txt and my_program.exe > output_2.txt) continue to execute.
4. The same BAT file from #1 gets executed again.
5. The new instance of my_program.exe > output_launcher.txt fails with the error “The process cannot access the file because it is being used by another process.”

I don't get the error if the BAT file from #1 does not redirect output, and I don't get the error if the two launched instances exit prior to executing the BAT file for a second time.

So, I'm assuming that CMD.EXE is holding exclusive rights to the output_launcher.txt file until all of the sub-processes are finished.

First of all, is that a good assumption?

What I would like to happen is that CMD.EXE gives up its rights to output_launcher.txt when the original instance exists in #3 since each sub-process is redirecting to its own file.

Is this possible when redirecting standard output? The best alternative I can think of would be to actually take in the log file location as a command line argument, and write to the file directly from my program, rather than redirecting standard output. This would involve a lot more work for me though, so I'd like to avoid that route if possible.

Thanks!

EDIT: The actual command line that the first instance uses to start the additional instances looks like start cmd /C "call my_program.exe > output_1.txt". Then I pass that command to the "system()" function (my_program.exe is written in MSDN C).

Maybe I could launch the additional instances in a different way that could help?

Answer 1

After some brief testing, I would say that your assumption seems to be correct. If you'd like to verify that, I recommend Process Hacker. (There is a "Downloads" button at the top; be careful as the ads are showing bad download links.) Go to Hacker, Find Handles or DLLs... (Ctrl-F)

Using that you can easily verify what is using the file, and by re-performing the lookup, can verify when the file is released.

I expect that Process Explorer could also do this similar.

Note that you may be better off writing to a unique filename... consider this: instead of output_1.txt the first sub-call, and output_2.txt the second sub-call, check what files pre-exist. Then go higher, so that you don't end up overwriting a file or bumping into problems because you cannot append because the file is in use. Operating systems and programming languages frequently provide functionality to generate a unique filename.

You might want to write to many temporary files, and then combine them into fewer files. Whether that idea makes a whole lot of sense, or none at all, may depend on your project and how you approach things.

If you have the source code to program.exe, instead of running "program.exe > filename", just use "program.exe filename". Place code in your program to write to the file quickly, and then close the file so that it is no longer being held up when there isn't an active write. (A more thorough process may even perform some "locking"... since you're running multiple copies of the program, that might actually be worth investing in.) Instead of calling printf(), use a custom function that only calls printf() if there is no output file specified, otherwise it writes to a file. Then, instead of needing to change all your printf() calls every time you change the output file, you simply need to change the value of one variable that gets passed to your custom output function. I know you said this might not be preferable due to needing to take some more time. However, I will let you know that I've personally saved myself a lot of time and effort after I came up with that approach. I view this is as the good long-term solution.

Or, consider using the operating system's logging functionality. This can be done by running a command from the command line: In Unix, run "logger"; in MS Windows, run EventCreate. There ought to be ways you can do that directly from your programming language choice (MSDN C, as you noted). That might be unideal if your output is lengthy, or if you're concerned with cluttering your OS logs. (I know some places have active monitoring of OS logs.)

Answer 2

It is a reasonable assumption.

1) Please verify that the file is not open in any other application. A reasonable way to do this would be to try to rename the files while the .bat application is not running. If this fails, some other application is holding the file open.

2) Can you try using the Append Operator >> instead of > and tell me what happens?