Keep Blender open after running script from command line for higher rendering throughput

Question

I am using Blender for on-the-fly data generation for a machine learning task. The data creation process should be fast, and is conditioned on some parameter values given by my model, e.g., an RGB color value.

The model training happens in Python. For now, I pass the created values to Blender, where I have a script called render, that reads the passed [args] and uses them to render an image via bpy.ops.render.render.

Python side:

os.system('blender -b testFile.blend --python-text render -- [args]') 

Blender side:

# retrieve args
...

# set scene params with args 
...

# render
bpy.context.scene.render.filepath = path
bpy.ops.render.render(write_still=True)

This works fine. The two main bottlenecks for speed are a) writing the image to (and reading it back from) disk, and b) opening and closing Blender every time. I can work around a) by setting up a RAM disk and writing/reading directly to/from that. But I don't know how to work around b).

Is it possible to always have Blender open in some background process, and then call its script render which reads my arguments and renders an image?

Answer 1

I found a neat way to do it. The trick is to have blender open as a non-blocking background process using the subprocess module, and then, when starting up Blender, create a server that listens on a local port. From the python script that executes the machine learning task, we can now send data to that port, which, upon arrival, will invoke the rendering process in Blender.

Python Script:

import time
import socket
from subprocess import Popen

# start non-blocking subprocess with blender in the background. 
# this will execute the script "render_server" in blender. 
process = Popen(['blender -b ../gradientTest.blend --python-text render_server'], shell=True)

# give blender some time to load and run the script that opens the server port.
# without this, we will get OSError[111]: Connection Refused.
time.sleep(1.0)

# set up the socket connection 
host = socket.gethostname()
port = 12347                   # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))

for j in range(2):
    # do arbitrarily complex things 
    ... 

    # string of arguments, will be parsed on the blender side.
    args = '1.0 0.0 0.0 0.0 1.0 0.0' if j == 0 else '1.0 1.0 1.0 0.0 0.0 0.0'
    s.sendall(str.encode(args))   # send to server running within blender 
    data = s.recv(1024)           # get ack back, to see that server is finished
    print('Received', data.decode('utf-8'))

print("Done. Closing.")
s.close()

Blender Script:

import bpy
import socket


host = ''        # Symbolic name meaning all available interfaces
port = 12347     # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))

s.listen(1)
conn, addr = s.accept()

while True:

    try:
        data = conn.recv(1024)

        if not data:
            print("no data, exiting.")
            break

        received_info = data.decode('utf-8')
        print("Client Says: ", received_info)
        vals = received_info.split(' ')         # split args string into list of numbers  
        render_images(vals)                     # render with those 
        conn.sendall(b'ack')                    # send back ack signal to acknowledge we're done 

    except socket.error:
        print("Error Occured. Exiting.")
        break

conn.close()