Questions tagged [prefect]
Prefect is a Python-based workflow management system (ETLs are an example use-case). Users organize Tasks into Flows, define dependencies, schedules, etc., and Prefect takes care of the rest.
prefect
199
questions
2
votes
1
answer
36
views
How can I run Prefect flows using a Docker Worker pool and a local custom Docker image?
I'm new to Prefect and have a limited experience with Docker and I'm trying to deploy some existing Prefect flows so that they run inside a Docker container running an image I built myself, but ...
- 558
0
votes
0
answers
38
views
Prefect is not running on Docker Container
I'm new to docker and prefect and trying to run a basic prefect flows with docker container from my local windows.
Here is my Docker file
# We're using the latest version of Prefect with Python 3.10
...
0
votes
0
answers
14
views
Prefect Doesn't Terminate Completed Flows in Windows
I came across the following situation:
When prefect finishes a task, it continues to run in the windows server task manager.
E.g.:
The change name flow ended with full status, but it is still running ...
0
votes
0
answers
47
views
Prefect worker using docker infrastructure not working properly after deploy
I'm trying to run a prefect worker on a work pool based on a docker infrastructure. I don't know why, it always gives me problems when the worker submits the flow and the container is run up. Maybe it ...
0
votes
0
answers
26
views
Prefect Database high Lock time
I am running prefect 2.7.1 on Kubernetes on AWS EKS.
I am using AWS RDS m7g.4xlarge as the database for prefect
I am able to see spike in the wait time in the performance monitoring of the RDS going ...
- 599
0
votes
0
answers
43
views
How do I get the Prefect "flow id" (uuid) from within a "task" in a programmatic way using Python?
How can I get the "flow id" (not the "flow run id") from Prefect using Python from within a "task?" The task is being executed by a flow. I'm using Prefect version 2.19.1 ...
- 23
0
votes
1
answer
117
views
Accessing the scheduled time from within a Prefect flow run
I plan to run Prefect 2 flows on a cron schedule - but is there any way to access, at runtime, the timestamp at which this particular run was scheduled?
For example, if the flow is scheduled to run ...
- 42.4k
0
votes
1
answer
79
views
Unable to run the Prefect server on machine's IP address when using NSSM
I have configured Prefect server on our Windows VM so that it runs on the machine's IP address. Here's how I did it.
env\Scripts\activate.bat
(env) prefect config set PREFECT_SERVER_API_HOST='SERVER-...
- 105
0
votes
0
answers
41
views
Prefect server :prefect.exceptions.ParameterBindError: Error binding parameters for function
i am attempting to use prefect to manage my flow,but adding @flow will result in an error.Can methods modified by @ flow not be related to classes?
enter image description here
Can methods modified ...
0
votes
1
answer
137
views
Dockerfile runs runs locally but fails on Google Cloud Run
I have a Dockerfile, shown below:
FROM --platform=linux/amd64 prefecthq/prefect:2-python3.12
RUN apt-get update && apt-get -y install libpq-dev gcc
RUN pip install --upgrade pip
RUN pip ...
- 1,535
0
votes
1
answer
373
views
Run docker image as prefect task or flow
My question is effectively a duplicate of this question however the only answer is related to Prefect 1. I would like to know if anyone is aware of how to do such a thing in Prefect 2.
I am able to ...
- 70
0
votes
1
answer
98
views
Prefect: Unknown work pool type 'docker'. Please choose from block, process
I am attempting to follow the tutorial in the Prefect docs and I have gotten to the section where I need to create a docker work pool using the following command:
prefect work-pool create --type ...
- 70
0
votes
1
answer
124
views
Prefect and aiosqlite dependency problem - sqlalchemy.exc.OperationalError: (sqlite3.OperationalError)
I'm running a script with prefect using some tasks and flows. This script has run without problems before, but i'm experiencing problems with it without having changed it.
script
I just call ...
0
votes
0
answers
143
views
ERROR | prefect.flow_runs.runner - Process for flow run exited with status code: -7
I am getting this Prefect error while I am running my script. The error is the following:
0%| | 0/2070 [00:00<?, ?it/s]09:36:01.917 | ERROR | prefect.flow_runs.runner - Process for flow ...
0
votes
0
answers
22
views
Training sklearn estimators on large datasets throws aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
I am trying to run prefect flow on Kubernetes. I have bigger dataset and want to train multiple estimators in parallel. See the below code for more in details.
from sklearn.datasets import ...
- 1