0

So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a distributed task queue, allowing users to lookup their job run statuses or computation results via ID of the saved model. However, I don't want this to be enqueued to a messaging queue just once, rather I want this saved job to run on a schedule, let's say once every hour.

The problem now is how do I get the job to be enqueued multiple times a day. I know I can use a cron job to read the entire jobs table, iterate them all, and wait for all of them to be pushed up to queue but that sounds like a faux pas that would inadvertently create the very problem a task queue attempts to avoid. Furthermore, I see a lot of folks putting their cron job straight into the same API server process which probably doesn't scale well in the same process. Even as a standalone, I don't see how you could feasibly partition the table and scale the cron process. Any suggestions on what is a good architecture or best practices?

Improve this question
3
  • "inadvertently create the very problem a task queue attempts to avoid" -- how exactly does having the CRON schedule push each of these jobs into a queue every hour cause a problem? It seems to me that this would solve the problem of re-scheduling and re-queueing while still having the work distributed evenly amongst the queue's consumers; what else is missing? Where does the need to scale the CRON process itself come from if all it's doing is pushing each job into a queue?
    – Ben Cottrell
    Commented May 15, 2023 at 6:47
  • @BenCottrell Well that’s exactly why I’m asking. There’s going to be some latency with reading the db for the jobs as well as sending the payload to the queue. Would it make sense to somehow partition the table and have multiple crown jobs in separate processes? I’m just not sure what the best practices are. I’ve seen people slap node cron in their app process and actually perform the long operations directly without a care in the world.
    – Malik Brahimi
    Commented May 15, 2023 at 10:38
  • What kind of latency numbers are you experiencing when reading the database and sending the payload to the queue? How are you measuring it? What's the maximum latency deemed acceptable by users/stakeholders of your application? (Keeping in mind that these jobs are only running once per hour, I'd find it strange if your stakeholders were affected by the time for querying a database and posting to a message queue, unless there's some other consideration here).
    – Ben Cottrell
    Commented May 16, 2023 at 8:18

2 Answers 2

Reset to default
3

In a comment on a previous answer you write that you consider the execution time of the cron job (reading 10k entries from database and enqueuing them) problematic. Did you actually measure the time and found an issue? If not, you're optimizing prematurely, which you should not do.

Unless there's significant processing involved, this cron job should need a few seconds at most.

Improve this answer
1
  • Ok sure thing, thanks for the advice
    – Malik Brahimi
    Commented May 16, 2023 at 0:44
0

You could use library like Quartz.NET

When request comes, you create a CRON job with desired timing setup. The library persists it in database and calls it appropriately. All the scaling isuses are then handled by the library.

You don't need messaging library, because all the execution would be handled by Quartz. And it shouldn't be a performance issue unless you are trying to run thousands of jobs a second.

Improve this answer
7
  • The job execution part is fine because I plan on having scaled worker services that consume a message queue or use something like hangfire. The real concern for me is the cron job because reading a DB table with 10k entries, iterating, and waiting for the latency of dispatching these jobs seems problematic
    – Malik Brahimi
    Commented May 15, 2023 at 5:07
  • And would you recommend creating this cron job in the same process as the API as many do in the express/node world or having a standalone console app with the cron job? If standalone, how would you scale that to chunk the DB table?
    – Malik Brahimi
    Commented May 15, 2023 at 5:08
  • Reading the Quartz docs right now ... is there a way to give precedence to the master cron job such that that thread continues to run as the child jobs are processed? i.e. I don't want the jobs to stop coming in while the threads have been maxed out
    – Malik Brahimi
    Commented May 15, 2023 at 5:10
  • 2
    I have used Quartz (JVM not .net) - it is a distributed scheduler allowing you to run quartz on multiple nodes - in such a configuration each execution can be run on any of the nodes (default behavior). Its your choice whether you want to embed that in process with your web/application servers or run it as a standalone service. I don't think there is a standalone process - you will need to wrap the scheduler with a lightweight rest service that simply provides an API for scheduling new jobs (which you would call from your main app servers).
    – DavidT
    Commented May 15, 2023 at 6:32
  • 2
    @MalikBrahimi I get the feeling that you are looking for certainty and confidence by asking strangers on the internet. The problem is that you won't find them until you actually put your hands on the dirt. We can't get rid of the uncertainty for you. You cannot skip the pain of the learning curve or the learnings that comes from failures. So, why don't you implement a proof of concept? Experimenting is the best way to get insights and a better perspective of what's possible or not. y
    – Laiv
    Commented May 15, 2023 at 13:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.