How to execute a python script in a different directory?

Question

Solved see my answer below for anyone who might find this helpful.

I have two scripts a.py and b.py. In my current directory "C:\Users\MyName\Desktop\MAIN", I run > python a.py.

The first script, a.py runs in my current directory, does something to a bunch of files and creates a new directory (testA) with the edited versions of those files which are simultaneously moved into that new directory. Then I need to run b.py for the files in testA.

As a beginner, I would just copy and paste my b.py script into testA and execute the command again "> python b.py", which runs some commands on those new files and creates another folder (testB) with those edited files.

I am trying to eliminate the hassle of waiting for a.py to finish, move into that new directory, paste b.py, and then run b.py. I am trying to write a bash script that executes these scripts while maintaining my hierarchy of directories.

#!/usr/bin/env bash
 python a.py && python b.py

Script a.py runs smoothly, but b.py does not execute at all. There are no error messages coming up about b.py failing, I just think it cannot execute because once a.py is done, b.py does not exist in that NEW directory. Is there a small script I can add within b.py that moves it into the new directory? I actually tried changing b.py directory paths as well but it did not work.

For example in b.py:

mydir = os.getcwd() # would be the same path as a.py
mydir_new = os.chdir(mydir+"\\testA")

I changed mydirs to mydir_new in all instances within b.py, but that also made no difference...I also don't know how to move a script into a new directory within bash.

As a little flowchart of the folders:

MAIN # main folder with unedited files and both a.py and b.py scripts
|
| (execute a.py)
|
--------testA # first folder created with first edits of files
         |
         | (execute b.py)
         |
         --------------testB # final folder created with final edits of files

TLDR: How do I execute a.py and b.py from the main test folder (bash script style?), if b.py relies on files created and stored in testA. Normally I copy and paste b.py into testA, then run b.py - but now I have 200+ files so copying and pasting is a waste of time.

Answer 1

The easiest answer is probably to change your working directory, then call the second .py file from where it is:

python a.py && cd testA && python ../b.py

---

Of course you might find it even easier to write a script that does it all for you, like so:

Save this as runTests.sh in the same directory as a.py is:

#!/bin/sh
python a.py
cd testA
python ../b.py

Make it executable:

chmod +x ./runTests.sh

Then you can simply enter your directory and run it:

./runTests.sh

Answer 2

I managed to get b.py executing and producing the testB folder where I need it to, while remaining in the MAIN folder. For anyone who might wonder, at the beginning of my b.py script I would simply use mydir = os.getcwd() which normally is wherever b.py is.

To keep b.py in MAIN while making it work on files in other directories, I wrote this:

mydir = os.getcwd() # would be the MAIN folder
mydir_tmp = mydir + "//testA" # add the testA folder name
mydir_new = os.chdir(mydir_tmp) # change the current working directory
mydir = os.getcwd() # set the main directory again, now it calls testA

Running the bash script now works!

Answer 3

In your batch file, you can set the %PYTHONPATH% variable to the folder with the Python module. This way, you don't have to change directories or use pushd to for network drives. I believe you can also do something like

set "PYTHONPATH=%PYTHONPATH%;c:\the path\to\my folder\which contains my module"

This will append the paths I believe (This will only work if you already have set %PYTHONPATH% in your environment variables).

If you haven't, you can also just do

set "PYTHONPATH=c:\the path\to\my folder\which contains my module"

Then, in the same batch file, you can do something like

python -m mymodule ...

Answer 4

despite there are already answers i still wrote a script out of fun and it still could be of help in some respects. I wrote it for python3, so it is necessary to tweak some minor things to execute it on v2.x (e.g. the prints).

Anyways... the code creates a new folder relative to the location of a.py, creates and fills script b.py with code, executes b and displays b's results and errors.

The resulting path-structure is: testFolder |-testA | |-a.py |-testB | |-b.py

The code is:

import os, sys, subprocess

def getRelativePathOfNewFolder(folderName):
    return "../" + folderName + "/"

def getAbsolutePathOfNewFolder(folderName):
    # create new folder with absolute path:
    #   get path of current script:
    tmpVar = sys.argv[0]
    #   separate path from last slash and file name:
    tmpVar = tmpVar[:sys.argv[0].rfind("/")]
    #   again to go one folder up in the path, but this time let the slash be:
    tmpVar = tmpVar[:tmpVar.rfind("/")+1]
    #   append name of the folder to be created:
    tmpVar += folderName + "/"

    # for the crazy ones out there, you could also write this like this:
    # tmpVar = sys.argv[0][:sys.argv[0].rfind("/", 0, 
    sys.argv[0].rfind("/")-1)+1] + folderName + "/"
    return tmpVar

if __name__ == "__main__":
    # do stuff here:
    # ...
    # create new folder:
    bDir = getAbsolutePathOfNewFolder("testB")
    os.makedirs(bDir, exist_ok=True) # makedirs can create new nested dirs at once. e.g: "./new1/new2/andSoOn"
    # fill new folder with stuff here:
    # ...
    # create new python file in location bDir with code in it:
    bFilePath = bDir + "b.py"
    with open(bFilePath, "a") as toFill:
        toFill.write("if __name__ == '__main__':")
        toFill.write("\n")
        toFill.write("\tprint('b.py was executed correctly!')")
        toFill.write("\n")
        toFill.write("\t#do other stuff")

    # execute newly created python file
    args = (
        "python",
        bFilePath
    )
    popen = subprocess.Popen(args, stdout=subprocess.PIPE)
    # use next line if the a.py has to wait until the subprocess execution is finished (in this case b.py)
    popen.wait()
    # you can get b.py´s results with this:
    resultOfSubProcess, errorsOfSubProcess = popen.communicate()
    print(str(resultOfSubProcess)) # outputs: b'b.py was executed correctly!\r\n'
    print(str(errorsOfSubProcess)) # outputs: None

    # do other stuff

instead of creating a new code file and filling it with code you of course can simply copy an existing one as shown here: How do I copy a file in python?

Answer 5

Your b.py script could take the name of the directory as a parameter. Access the first parameter passed to b.py with:

import sys
dirname = sys.argv[1]

Then iterate over the files in the named directory with:

import os
for filename in os.listdir(dirname):
    process(filename)

Also see glob.glob and os.walk for more options processing files.