Questions tagged [python-multiprocessing]
multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.
python-multiprocessing
4,888
questions
0
votes
0
answers
13
views
Why does mp.Process.is_alive return True even after the process has finished [closed]
OS: Linux localhost.localdomain 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Python: 3.7.17
I used mp.Process.is_alive in the child thread to wait for the ...
0
votes
1
answer
31
views
OpenCV GStreamer VideoWriter write hangs in a Python multiprocessing Process
I'm working on real-time image processing using a Raspberry Pi camera. To optimize processing times, I'm trying to separate some image transformations and video encoding in a separate Process:
import ...
0
votes
0
answers
37
views
Why I am facing [Errno 32] Broken pipe when I start new process?
I am using confluent-kafka to process messages from kafka topic.
Here is the code template:
from multiprocessing import Process, Manager
def handle_records(records):
for record in records:
...
- 53
-3
votes
0
answers
18
views
How to efficiently reuse the same img_path against sets of photos? [closed]
How can supply img_path in an efficient way to do multiple searches of the same photo against different photo sets without having to do recomputations for img_path each time find is called?
Something ...
- 1
0
votes
1
answer
64
views
What is the closest thing to Python multiprocessing's Pool.imap when the iterator only works on the main thread?
Suppose I have an iterator that only works on the main thread (throws an exception otherwise), but I still want to distribute work (one task per item from the iterator) over several processes. (...
- 213
1
vote
1
answer
61
views
Trouble Capturing All Tick Data with Concurrent.futures in Python [closed]
I am developing a program to insert tick-level data for 300+ stocks received in a webstream dictionary format from a broker API. I successfully have a version of the program running without concurrent....
2
votes
0
answers
30
views
Python Multiprocessing with Pipe: Subprocesses sleep forever when training deep reinforcement learning agent with no error raised
I was using Pytorch-based Deep Reinforcement Learning (DRL) agent to train the environment with Multiprocessing package to train in multiple processes (thanks to the renowned elegantrl framework).
To ...
- 21
0
votes
0
answers
25
views
ThreadPoolExecutor slows down while running
I have a list of arguments that can be executed in any order. I have written one function to parse and write to an S3 bucket. The function output is a CSV file that's written to the bucket.
def func(...
0
votes
1
answer
28
views
Python multiprocessing and how to wrap up and successfully exit
I'm learning python as I go and am currently implementing multiprocessing but I cannot successfully exit the app I've written. It seems the 'capture_read_frame_process' process just doesn't end and I ...
- 11
0
votes
1
answer
43
views
Python: Logging to a single file when having a ProcessPoolExecutor within Process workers, using QueueHandler
I want to add logs to a single file from different Processes, which contain a ProcessPoolExectuor.
My application has the following strcture:
Main Process
Secondary Process with a pool of processes: ...
- 1,053
1
vote
1
answer
17
views
How can a Python Ctypes Structure inside Multiprocessing Value cause a Recursion Error?
I'm trying to share a Ctypes Structure between some processes using Multiprocessing.Value(). For some reason, this one struct isn't working, and causes a random RecursionError somewhere deep inside ...
- 445
0
votes
0
answers
21
views
Multiprocessing with gramformer
I am trying to implement multiprocessing with gramformer which is an open source model to correct grammatical errors. I have tried several approaches but i keep getting errors cant be pickled. i am ...
- 11
0
votes
1
answer
105
views
Polars - issues with performance - attempting to create a new dataframe per row
I need to run another library's algo on each row of a large df but am having trouble converting my code to polars expressions for better performance. Here are a couple sample DFs:
df_products = pl....
0
votes
0
answers
26
views
How do I properly terminate a python multiprocessing process that runs a GStreamer pipeline
I am running a GStreamer pipeline in a separate Python multiprocessing process. The process captures frames from a camera to a multiprocessing Queue (full_frame_queue). I want to stop the process ...
- 35
1
vote
0
answers
52
views
How to spawn a child process in a python-embedded frozen application?
As far as I know, in python multiprocessing module, the entry function of a child process is the function: multiprocessing.spawn.spawn_main.
According to the following function (multiprocessing.spawn....
- 21