All Questions
Tagged with concurrency python
1,336
questions
0
votes
1
answer
25
views
Recursive use of ProcessPoolExecutor is Hanging
Problem
I'm trying to use a ProcessPoolExecutor with recursive calls, but it doesn't work. I created a minimal example below
from concurrent.futures import ProcessPoolExecutor
from time import sleep
...
- 159
0
votes
0
answers
11
views
Concurrency Control Mechanism For Dataframe Processing In Django WebApp
I have django webapp where processing data directly on pandas dataframe without using django model. now, i want to make this operations concurrency control for multiple processing requests ...
0
votes
0
answers
18
views
Streamlit: keep fragment running while the page loads
I have a Streamlit UI with this code
import streamlit as st
import time
_LOREM_IPSUM = """
Lorem ipsum dolor sit amet, **consectetur adipiscing** elit, sed do eiusmod tempor
incididunt ...
- 191
0
votes
0
answers
25
views
ThreadPoolExecutor slows down while running
I have a list of arguments that can be executed in any order. I have written one function to parse and write to an S3 bucket. The function output is a CSV file that's written to the bucket.
def func(...
0
votes
1
answer
33
views
Concurrently test several Pytorch models on a single GPU slower than iterative approach
I want to test several models on different datasets. I want them to run concurrently on a single gpu.
The general puesdo code is below. You would be able to run this with any dummy model or dataset.
...
- 37
1
vote
0
answers
32
views
Optimizing Custom Double Release Lock and Semaphore in Python asyncio
I am currently working on a project involving file operations that need to be handled by multiple processors concurrently, but not by other threads simultaneously. For this, I've devised a concept ...
- 21
0
votes
1
answer
33
views
Facing issue with flask server where one of the endpoints writes to a common file
For some reasons, I cannot post the actual code; I will try my best to explain the issue I am facing clearly.
In my dev setup, I am running ansible play to configure 3 or more nodes.
Also, locally I ...
- 343
0
votes
1
answer
23
views
Does Ray Offer any Functional/Declarative Interface to Map a Remote Function to an Iterator/Iterable?
My present Code
#!/usr/bin/env python3
# encoding: utf-8
"""Demonstration of Ray parallelism"""
import ray
from typing import Iterator
ray.init()
@ray.remote
def square(n:...
- 1,560
0
votes
0
answers
10
views
Add locks for timed kv store
I am trying to add lock to timed key value store with python. Following is my code:
class TimedKVWithLock:
def __init__(self):
self.data = {}
self._locks = defaultdict(threading....
- 11
0
votes
0
answers
35
views
TimedRotatingFileHandler thread safe in high concurrency env (fastapi with gunicorn)
Context
I have a python API with dozen of routes. I created a custom logger to log every I/O of my app. The file which is the destination of my TimedRotatingFileHandler is missing a lot of logs that ...
- 103
0
votes
0
answers
78
views
How to Optimize Python's Multithreaded Performance for Real-Time Machine Learning Predictions?
I'm currently developing a real-time machine learning application using Python, and I'm facing challenges with optimizing multithreaded performance to reduce latency in model predictions. Even with ...
- 83
1
vote
2
answers
150
views
Multithreading vs multiprocessing in I/O bound processes
I was reading an article about concurrency in python and decided to try the code out in my local machine. If I understand it correctly then multithreading is supposed to be better at I/O bound ...
0
votes
0
answers
12
views
Celery graph only returns the first group executed
tl;dr: A chain of groups and tasks executes correctly in Celery, but the result backend only returns the result of the first group. How do I fix this?
Here is my code:
app.task
def add(prev_result, x, ...
- 4,648
1
vote
2
answers
32
views
How to use ThreadPoolExecutor concurrently with infinite loops
I'm developing a application to fetch data from different servers that are bound to each other.
Basically, each server has a major region, like America or Asia, and sub regions, like North America, ...
2
votes
1
answer
27
views
Python giving different response when called via async
Getting different output for the same logic?
Was trying to learn async/await in python.
Output for first Snippet : <class 'dict'>
Output for second Snippet : <class 'list'>
Snippet 1 : ...
- 65